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Key findings 
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Data specialists in the state of Chuuk in the Federated States of Micronesia 
assessed their education management information system using a rubric 
with four benchmark levels that rated five aspects of system quality with 46 
indicators. The four benchmark levels were latent (not in place), emerging (in 
the process of implementation), established (in place and meeting standards), 
and mature (an example of best practice). The overall system was rated as 
emerging. Three of the five aspects of system quality also received that rating: 
Integrity of education statistics. 

Serviceability. 

Accessibility of data to stakeholders. 

Two aspects of quality were rated as established: 

Prerequisites of quality. 

Accuracy and reliability of education statistics. 

The report also provides the scores for the 46 indicators that were used to 
calculate the benchmark level of the system overall and the five aspects of quality. 


ries 


NATIONAL CENTER for 
EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 




Institute of Education Sciences 
U.S. Department of Education 



U.S. Department of Education 

John B. King, Jr., Secretary 

Institute of Education Sciences 

Ruth Neild, Deputy Director for Policy and Research 
Delegated Duties of the Director 

National Center for Education Evaluation and Regional Assistance 

Joy Lesnick, Acting Commissioner 
Amy Johnson, Action Editor 

Chris Boccanfuso and Janelle Sands, Project Officers 
REL 2017-176 

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts 
unbiased large-scale evaluations of education programs and practices supported by federal 
funds; provides research-based technical assistance to educators and policymakers; and 
supports the synthesis and the widespread dissemination of the results of research and 
evaluation throughout the United States. 

October 2016 

This report was prepared for the Institute of Education Sciences (IES) under Contract 
ED-IES-12'C'OOIO by Regional Educational Laboratory Pacific administered by McREL 
International. The content of the publication does not necessarily reflect the views or pol- 
icies of IES or the U.S. Department of Education, nor does mention of trade names, com- 
mercial products, or organizations imply endorsement by the U.S. Government. 

This REL report is in the public domain. While permission to reprint this publication is 
not necessary, it should be cited as: 

Kendall, J. S., Dandapani, N., & Cicchinelli, L. F. (2016). Benchmarking the state of Chuuk’s 
education management information system (REL 2017-176). Washington, DC: U.S. Depart- 
ment of Education, Institute of Education Sciences, National Center for Education Evalu- 
ation and Regional Assistance, Regional Educational Laboratory Pacific. Retrieved from: 
http://ies.ed.gov/ncee/edlabs. 

This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/ 
ncee/edlabs. 



Summary 


A quality data management system, such as an education management information system 
(EMIS), a state longitudinal data system, or a data warehouse, is key to ensuring that edu- 
cation policy, planning, and strategy decisions are grounded in accurate information (Data 
Quality Campaign, 2010; Mohamed, Kadir, May-Lin, Rahman, & Arshad, 2009; Regional 
Educational Laboratory Southeast, 2010; World Bank, 2011). The chief state school officers 
of the Federated States of Micronesia have recognized the need to improve their national 
EMIS and those of their four states, especially the organization of data and timeliness of 
data reports (Federated States of Micronesia, 2009). 

For this study, Regional Educational Laboratory (REL) Pacific assessed the EMIS in the 
state of Chuuk by convening a focus group of data specialists to rate the processes, tools, 
and materials that support the ability to provide timely and meaningful data to schools or 
education agencies. They used a tool developed by the World Bank to objectively assess 
and monitor the quality of EMISs. The tool assesses five major aspects of quality: pre- 
requisites of quality, integrity of education statistics, accuracy and reliability of education 
statistics, serviceability (relevance, timeliness, and consistency of data), and accessibility 
of data to stakeholders. Scores from the data specialists were averaged and used to assign 
each aspect of quality to one of four benchmark levels: latent (not in place), emerging (in 
the process of implementation), established (in place and meeting standards), and mature 
(an example of best practice). 

The results of this study provide a comprehensive view of Chuuk’s EMIS, which overall 
was rated as emerging. Three of the five aspects of quality were also rated as emerging: 

• Integrity of education statistics (the professionalism, objectivity, transparency, and 
ethical standards by which staff operate and statistics are reported). 

• Serviceability (relevance, timeliness, and consistency of data). 

• Accessibility of data to stakeholders. 

The remaining two aspects were rated as established: 

• Prerequisites of quality (the institutional frameworks that govern the information 
system and data reporting and the supporting resources). 

• Accuracy and reliability of education statistics in the system. 
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Why this study? 


The chief state school officers in the Federated States of Micronesia have called for improv- 
ing the education management information system (EMIS) nationally and in the country’s 
four states (Chuuk, Kosrae, Pohnpei, and Yap), especially in the collection and organi- 
zation of data and the timeliness of data reports (Federated States of Micronesia, 2009). 
An EMIS encompasses all elements related to the collection, storage, and processing of 
data to formulate, manage, and evaluate education management policies. A 2010 audit by 
the Federated States of Micronesia National Department of Education underscored the 
need for better data communication systems — the networking, delivery, and reporting of 
data within the national department and between it and the states. The audit also recom- 
mended that the national and state departments of education improve the uniformity and 
accuracy of data collection (Wrembeck & Fenlon, 2010). 

In Chuuk, as well as in the other states, education data are critical for demonstrating 
compliance with programmatic and grant funding requirements. A 2011 grant award to 
the Federated States of Micronesia’s education sector from the U.S. Department of the 
Interior required that the country develop a comprehensive strategy to improve centralized 
data collection and analysis (Joint Economic Management Committee, 2010, Resolution 
2010-6; U.S. Department of the Interior, 2010). The Joint Economic Management Com- 
mittee, which oversees the allocation and use of funds from the U.S. Department of the 
Interior’s Office of Insular Affairs Compact, has the authority to withhold funds from the 
national or state departments of education if it determines that system performance or 
improvement is unsatisfactory. 

The performance measures established for the Federated States of Micronesia’s EMIS were 
initially created in response to a 2004 Joint Economic Management Committee resolution 
that called for developing appropriate performance measures “for all sectors with base- 
line data that are consistent over time, verifiable, understandable and easy to report and 
monitor” (Joint Economic Management Committee, 2004, pp. 9-10; see appendix A for 
a review of the literature on EMISs). Since 2005 the Joint Economic Management Com- 
mittee has required the Federated States of Micronesia National Department of Education 
to submit an annual report summarizing data across the Federated States of Micronesia’s 
four states on 21 education indicators. (The original number of indicators was 20; one was 
added in 2011 based on an Office of Insular Affairs recommendation to include reporting 
on scholarship data; see Federated States of Micronesia, 2013.) However, the most recent 
audit, contracted by the National Department of Education in 2010, noted that no formal 
data collection or verification policies or procedures were in place at the national or state 
levels, suggesting a need to establish minimum standards to strengthen data management, 
collection, and verification. The audit recommended that the National Department of 
Education “implement a standardized data information collection system across all educa- 
tion departments to allow for uniformity and accuracy of data collection” (Wrembeck & 
Fenlon, 2010, p. 11). 

It is in this context that Regional Educational Laboratory (REL) Pacific undertook this 
assessment of the quality of Chuuk’s EMIS. The findings will inform a plan of action with 
specific and measurable goals to improve the system over the near and long terms. This 
study complements an effort by the Federated States of Micronesia’s National Department 
of Education to satisfy the requirements of the Joint Economic Management Committee 
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resolution. Assessment and benchmarking of the EMISs in the three other states — Kosrae, 
Pohnpei, and Yap — have been conducted using the same protocol, and the results will be 
available in separate reports. 1 REL Pacific will also produce a summary report that explores 
the similarities and differences in the quality of EMISs across all four states. 

What the study examined 


To examine the quality of Chuuk’s EMIS, REL Pacific convened a focus group of eight 
Chuuk data specialists. During a one-day session in July 2014, they discussed and rated the 
processes, tools, and materials they use to collect, analyze, and report education data to 
provide timely and meaningful information to schools and education agencies (see box 1 
and appendix B for data sources and methods). These data specialists — nominated by the 
Chuuk director of education for their familiarity with the state’s data system — included a 
department chief, a transition specialist, statistics and assessment specialists, data clerks, 
and an individual working closely with outer island schools. Although just eight data 
specialists provided information for this study, they were the specialists with the most 
knowledge of the day-to-day operations of data collection, statistical analysis, and report 
dissemination for the state of Chuuk’s education department, which comprises 115 staff 
and 777 teachers. 

To examine the quality of Chuuk’s EMIS, REL Pacific adapted a tool developed by the 
World Bank that allows identification of the characteristics of an EMIS and comparison 
of systems within a country over time or across countries (see box 2 and appendix B). The 
System Assessment and Benchmarking for Education Results (SABER) EMIS Assessment 
Tool covers five aspects of quality of the EMIS and provided a benchmark system to rate its 
strengths and weaknesses (see box 2 and appendix C). 
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The five aspects of quality are prerequisites of quality (including the legal and institutional 
environment and whether human and material resources are adequate for the task), integ- 
rity of education statistics (including whether statistical policies and practices are guided 
by professional principles, whether they are transparent, and whether they are guided by 
ethical standards), accuracy and reliability of education statistics), serviceability (relevance, 
timeliness, and consistency), and accessibility (education statistics are made available in 
clear and understandable ways to stakeholders). 


Aspects of quality are rated at one of four benchmark levels: latent (not in place), emerg- 
ing (in the process of implementation), established (in place and meeting standards), and 
mature (an example of best practice; table 1). The protocol was revised slightly to make it 
more relevant to this study (see box 2). 
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Box 1. Data sources and methods 

Data sources. The data for this study are based on eight data specialists’ responses to ques- 
tions related to 46 indicators covering five aspects of the quality of Chuuk’s education man- 
agement information system (EMIS; see box 2). Each question was developed by Regional 
Educational Laboratory Pacific to help focus discussion on an indicator of system quality from 
the World Bank’s System Assessment and Benchmarking for Education Results (SABER) EMIS 
Assessment Tool. 

Data collection. Each data specialist received a printed copy of the indicators and related 
questions, scoring rubric, and accompanying explanatory text before a focus group discus- 
sion (see appendix B). Data specialists also had the opportunity to consider the questions in 
advance and to bring any evidence — regulations, logs, forms, and other artifacts — that sup- 
ported their perspective on how the indicators should be rated. After the facilitator presented 
each question, data specialists discussed it, and each rated the rubric on a five-point scale. 
Unanimity of response was not required. Data specialists were asked to review documentation 
of their responses and confirm that their expressed views were accurately reflected. 

Methodology and analysis. The responses for each indicator were analyzed using descriptive 
statistics, identifying the high, low, and average scores of the data specialists. The related 
indicator scores were averaged to obtain scores for the five aspects of quality (see box 2). 
The scores for each aspect were assigned the appropriate benchmark level: latent, emerging, 
established, or mature (see table 1 for a description of the levels). The aspect scores were 
then averaged to obtain an overall score. 


Box 2. The revised System Assessment and Benchmarking for Education Results 
Education Management Information System Assessment Tool 

The indicators and scoring guidelines used in this study were developed by the World Bank 
(2011) as part of the System Assessment and Benchmarking for Education Results (SABER) 
Education Management Information System (EMIS) Assessment Tool. Built from an Internation- 
al Monetary Fund framework to compare scores across countries, the tool was developed to 
assess and monitor EMISs on specific aspects of quality. The study team added a focusing 
question for each indicator to help guide the group discussions and dropped one aspect of 
quality measured by the tool (methodological soundness) because its focus on international 
standards did not align to this study’s purpose of comparing data across the Federated States 
of Micronesia (for a discussion, see appendix B). The revised tool examined five aspects of 
quality by scoring 7-11 indicators under each aspect (46 indicators in all). Based on those 
scores, each indicator, each aspect, and the overall system are then assigned to one of four 
benchmark levels. 

Five aspects of quality. The five aspects of quality measured by the revised SABER EMIS 
Assessment Tool are: 

1. Prerequisites of quality. Assesses whether the institutional frameworks that govern the 
information systems and data reporting and the supporting resources are available and 
adequate. This aspect has two subsections: the legal and institutional environment and 
whether human and material resources are adequate for the task. (11 indicators) 

(continued) 
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Box 2. The revised System Assessment and Benchmarking for Education Results 
Education Management Information System Assessment Tool (continued) 

2. Integrity of education statistics. Assesses whether the professionalism, objectivity, trans- 
parency, and ethical standards by which data staff operate and by which the data and sta- 
tistics are reported are adequate. This aspect has three subsections: whether statistical 
policies and practices are guided by professional principles, whether statistical policies 
and practices are transparent, and whether policies and practices in education statistics 
are guided by ethical standards. (Nine indicators) 

3. Accuracy and reliability of education statistics. Assesses whether the data and statistical 
techniques are sound and whether statistical reports reflect reality. (10 indicators) 

4. Serviceability (relevance, timeliness, and consistency). Assesses whether the data being 
collected and the education statistics being generated are relevant and available to inform 
policy and practice decisions and are consistent over time to measure progress. (Seven 
indicators) 

5. Accessibility. Assesses whether the education statistics (and their underlying metadata) 
are made available in clear and understandable ways to stakeholders. (Nine indicators) 

Indicators. Each indicator focuses on a specific characteristic of an aspect of quality; for 
example, one indicator under the serviceability aspect concerns whether statistics are released 
on a pre-announced schedule. Indicators are supplemented with a focusing question and clar- 
ifying statements ranging from a few sentences to several paragraphs that provide examples 
or context for the indicator. Indicators for aspects 1 and 2 are organized under subsections; 
indicators under aspects 3-5 are organized directly under their aspects (see appendix C). The 
question for each indicator in the focus group protocol can be answered on a five-point scale 
from 0 to 1. 

Benchmark levels. The score for each indicator can be placed within the range of scores that 
defines one of the four benchmark levels: latent, emerging, established, and mature (see 
table 1 in the main text for descriptions of the benchmark levels and the score ranges; see 
table B1 in appendix B for the score level for each aspect of quality). The scores for each indi- 
cator within an aspect of quality were averaged across data specialists to provide a sense of 
relative strengths and weaknesses within the aspect. The indicator scores were then averaged 
and used to assign the aspect to a benchmark level. The overall score for the EMIS was calcu- 
lated by averaging the scores for all five aspects and assigning it to the applicable benchmark 
level. For graphs of the average scores by indicator and aspect, see appendix D. 
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Table 1. Descriptions of benchmark levels for indicator scores and overall score 
range for aspects of quality 

Benchmark 

level 

Description 

Indicator 

score 

Aspect of quality and 
overall score range 

Latent 

The process or action required to improve the aspect of 
quality is not in place 

.00 

.00-.30 

Emerging 

The process or action is in the process of 
implementation 

.25 and .50 

.31-.59 

Established 

The process or action is in place and it meets standards 

.75 

.60-. 79 

Mature 

The process or action is an example of best practice 

1.00 

.80-1.00 

Source: Adapted from World Bank (2011), pp. 40 and 46. 




What the study found 


Data specialists in the Federated States of Micronesia state of Chuuk rated the state’s EMIS 
overall as emerging, meaning that it is in the process of implementation. The prerequisites 
of quality (the institutional frameworks that govern the system and data reporting and 
the supporting resources) were rated as established. The integrity of education statistics 
was rated as emerging. The accuracy and reliability of education statistics received the 
highest rating of all five aspects, within the established benchmark range. Serviceability 
(relevance, timeliness, and consistency) and the accessibility of education data to stake- 
holders were rated as emerging. 
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Chuuk’s education management information system overall was rated as emerging 

Data specialists rated Chuuk’s EMIS overall as emerging, based on a simple average of 
aggregate scores for all five aspects of quality (the average score was .58, in the emerging 
benchmark range of .31 — .59; figure 1). The SABER EMIS Assessment Tool describes the 
emerging benchmark level as: 

Basic data channels in place though still weaknesses in reporting by providers; 
some commitment to data use; data are still fragmented across ministries; coverage 
and relevance needs large improvement; some regional benchmarks used (World 
Bank, 2011, p. 48). 

The prerequisites of quality aspect was rated as established (with a benchmark level of 
.60—79), meaning that it is in place and meets standards. This aspect, which concerns the 
institutional frameworks that govern the systems and data reporting and the availability 
and adequacy of supporting resources, was scored an average of .64 across 11 indicators. 
The integrity of education statistics aspect was rated as emerging, with an average of .54 
across the nine indicators. The system’s accuracy and reliability was the highest-rated 
aspect, with an average rating of .65 across 10 indicators, which puts it into the established 
range. The last two aspects, serviceability and accessibility, were scored within the emerg- 
ing benchmark range (.56 and .48, respectively), indicating that the associated processes 
are in the process of implementation. 

The scores assigned by data specialists to each indicator are discussed by aspect of 
quality in the following sections, with a focus on indicators that received the highest and 
lowest scores and a discussion of how the average score of indicators for each aspect is 
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Figure 1. Chuuk’s education management information system overall was rated as 
emerging, and its highest-rated aspect was accuracy and reliability, 2014 


Quality aspect 

Prerequisites of quality 

Integrity 

Accuracy and reliability 

Serviceability 

Accessibility 


.00 .25 .50 .75 1.00 

Latent Emerging Established Mature 

. 00-. 30 . 31-. 59 . 60-. 79 . 80 - 1.00 

Source: Authors' calculations based on participant responses to the revised System Assessment and Bench- 
marking for Education Results focus group protocol in 2014. 


characterized within the revised SABER EMIS Assessment Tool. See appendix D for the 
range of individual scores for these indicators by aspect. 
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Both subsections of the prerequisites of quality aspect were rated as established 

Data specialists rated the prerequisites of quality aspects as established, with an average 
score of .64 (see figure D1 in appendix D). The SABER EMIS Assessment Tool describes 
the established benchmark level for this quality aspect as: 

Data sharing and coordination in place; confidentiality assured; legal framework 
indirectly in place through the legal mandate of the census agency; more efficien- 
cy in resource use is needed; users’ needs assessed but infrequently; data quality 
processes in place but enforcement needs improvement (World Bank, 2011, p. 46). 

The prerequisites of quality aspect includes 11 indicators in two subsections: legal and 
institutional environment and whether human and material resources are adequate for 
the task. The subsection on legal and institutional environments, which addresses the 
degree to which these environments support educational statistics, was rated as emerg- 
ing (-44)- The indicator rated lowest, both within this subsection and within this quality 
aspect, was on whether the responsibility for collecting and disseminating education data 
is clearly specified (.25, in the latent benchmark range of .00-30). The subsection regard- 
ing whether human and material resources are adequate for the task was rated as estab- 
lished (.76). Rated highest in this subsection and in the entire aspect was the indicator 
on whether mechanisms exist for addressing new and emerging data requirements (1.00, 
mature). 
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Integrity of education statistics was rated as emerging 


Data specialists rated the integrity of education statistics aspect as emerging, with an 
average score of .54 (see figure D2 in appendix D). The SABER EMIS Assessment Tool 
describes the emerging benchmark level for this aspect as: 

Statistics are not impartial; professionalism of the staff is promoted on a limited 
basis; errors in statistics are corrected sporadically; statistics have institutional 
backing but chain of responsibility is unclear; major changes in methods, source 
data and techniques are sporadically made public and guidelines for staff behavior 
are short and incomplete (World Bank, 2011, p. 47). 

The integrity of education statistics includes nine indicators in three subsections: whether 
statistical policies and practices are guided by professional principles, whether they trans- 
parent, and whether they are guided by ethical standards. 

Whether statistical policies and practices are guided by professional principles was rated as 
established, with an average score of .65. The indicator on whether statistics are produced 
on an impartial basis received the lowest rating in this subsection (emerging, at .40). 

Whether statistical policies and practices are transparent was rated as emerging, with 
an average score of .51. The clear identification of agency products was the highest-rated 
aspect, at the established level (.67). The lowest-rated indicator (.42, emerging) was one 
that addresses whether advanced notice is given regarding major changes in methodology, 
source data, and statistical techniques. 

Finally, whether ethical standards guide statistical techniques and decisions about dissent- 
ination was rated as latent. This subsection’s single indicator — whether guidelines for staff 
behavior are in place and are well known to staff — received a score of .25, the lowest indi- 
cator score in this aspect. 

The accuracy and reliability of education statistics aspect was rated as established 

Data specialists rated the accuracy and reliability aspect as established, with an average 
score of .65 (see figure D3 in appendix D). The SABER EMIS Assessment Tool describes 
the established benchmark level for this aspect as: 

Source data are assessed regularly; source data come from comprehensive data col- 
lection tailored to the country’s condition; the compilation of source data followfs] 
international standards; intermediate results are validated with other data sources 
if discrepancies are easily noted; statistical discrepancies are investigated and cor- 
rected regularly (World Bank, 2011, p. 47). 

The accuracy and reliability aspect comprises 10 indicators. Data specialists rated their 
system as established on all but two indicators, which were rated as emerging. One indi- 
cator, scored at .54, concerns whether source data are timely — specifically, whether source 
data agencies are compliant with deadline needs of education statistics. The other indi- 
cator, scored at .50, concerns whether source data are routinely audited for coverage and 
accuracy. Among those indicators rated established, the highest rated (at .79) concerned 
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whether source data reasonably approximate the definitions, scope, classifications, and 
time of recording required. 

The serviceability aspect was rated as emerging 

Data specialists rated the serviceability aspect as emerging, with a score of .56 (see figure 
D4 in appendix D). The SABER EMIS Assessment Tool describes the emerging bench- 
mark level for this aspect as: 

Statistics are published at regular intervals but their timeliness is inconsistent with 
international standards; statistics are sometimes inconsistent within the data set 
and with other data sources; revisions are sporadic and preliminary data are rarely 
identified (World Bank, 2011, p. 48). 

The serviceability aspect concerns whether education statistics are relevant, delivered in 
a timely manner, and consistent. This aspect also encompasses any revision policies that 
may affect the relevance, timeliness, or consistency of statistics. The highest-rated indi- 
cator (mature, .96) was the regularly scheduled dissemination of education statistics. An 
indicator on whether statistics are consistent or reconcilable with those obtained through 
other data sources was the lowest-rated indicator across all quality aspects (latent, .07). 
Next lowest in this aspect, scoring as emerging (.32), was an indicator on whether revisions 
follow a regular and established schedule. 

The accessibility of data to stakeholders aspect was rated as emerging 

Data specialists rated the accessibility of data to stakeholders aspect as emerging, with an 
average score of .48 (see figure D5 in appendix D). The SABER EMIS Assessment Tool 
describes the emerging benchmark level for this aspect as: 

The presentation of statistics needs improvement for better interpretation; dissem- 
ination is adequate but sporadic; documentation is deficient; level of detail con- 
forms to user needs, but there is no list of publications available to users (World 
Bank, 2011, p. 48). 

Related to this aspect was whether the statistics that the department of education makes 
available are clear and understandable to users and stakeholders. Presentation of statistics 
to facilitate proper interpretation and meaningful comparison received the highest score 
(mature, .82). 

An indicator addressing whether catalogs of publications and other services are widely avail- 
able received the lowest score (latent, .11); an indicator regarding whether levels of detail are 
adapted to the needs of the intended audience had the next-lowest score (emerging, .36). 

Implications of the study 


The results of this study provide the Chuuk State Department of Education and the Fed- 
erated States of Micronesia National Department of Education with information regarding 
the strengths of the Chuuk EMIS and the weaknesses that may benefit from immediate 
and focused improvement efforts. As noted, REL Pacific is also conducting studies on 


Data specialists 
rated the 
serviceability 
aspect and the 
accessibility 
of data to 
stakeholders 
aspect as emerging 


8 


the EMISs in Kosrae, Pohnpei, and Yap and will publish the findings in separate reports 
along with a summary report that synthesizes the findings for all states. These companion 
reports are intended to allow for further comparison of strengths, areas for improvement, 
and possible next steps that might be applied consistently across states. 


The data specialists rated Chuuk’s EMIS as established on two aspects, prerequisites of 
quality and system accuracy and reliability. On the three other aspects, and overall, the 
system was rated as emerging. From the perspective of system improvement, these ratings 
indicate that the institutional frameworks and resources that support the system, as well 
as accuracy and reliability of statistics, meet standards. Although these areas may be 
improved, the specialists’ ratings indicate greater concern and attention for internal poli- 
cies, practices, services, and products provided to agencies and schools. 

Lower ratings on the aspect of integrity of education statistics suggest that department 
data-related policies, standards, and practices would benefit from in-depth review by the 
department’s leadership and systems design experts. These policies and standards encom- 
pass both statistical policies and practices and ethical standards. The World Bank (2011) 
identifies integrity as crucial to ensuring that the general public can trust education data 
and trust that such data are not swayed or influenced by political forces. 

The aspects of serviceability and accessibility have a common focus in that they address 
how well schools and agencies are served by their EMIS. A rating of emerging indicates 
that the relevance, timeliness, and consistency of the data and statistics provided to stake- 
holders (serviceability) could be improved. The implications of these findings become 
clearer at the indicator level. For example, of significant concern, based on data special- 
ists’ scoring, is that statistics may not be consistent or reconcilable with those obtained 
through other data sources. Closely related to serviceability is accessibility — that data and 
statistics, once communicated to stakeholders, should be clear and readily understandable. 
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As a specific example of an area for improvement, detail in reports could be better adapted 
for the intended audience. Accessibility also concerns the timeliness with which data are 
made available to all users; indicators related to this topic (5.3 and 5.4; see appendix C) are 
among the lowest rated for this aspect. This may be of special concern because, as noted 
in the research literature, obsolete data have limited value for guiding decisionmaking and 
may result in missed opportunities for intervention and improvement (Hua & Herstein, 
2003). This rating comports with, and may help clarify, findings from the Federated States 
of Micronesia National Department of Education audit (Wrembeck & Fenlon, 2010): that 
collection and processing of data need improvement. 

The research literature suggests that core characteristics of the EMIS, including the collect- 
ing, integrating, and organizing of data and statistics, will be strengthened by establishing 
clear roles and responsibilities so that data specialists receive the data they require to provide 
timely reports. Clarification of those roles — and, more broadly, development of standardized 
data collection and verification policies and procedures — requires the coordinated efforts of 
state and national leaders. Such “coordination and collaboration at all levels in the educa- 
tion system” is another essential characteristic of an effective EMIS (Cassidy, 2006, p. 17). 

The scores from the data specialists should help focus efforts to improve system quality. 
The findings may be used to support further development of system performance measures 
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as recommended by the Joint Economic Management Committee by acting as baseline 
data for all sectors of the EMIS or by suggesting areas of investigation. For example, data 
specialists may want to investigate whether the noted lack of clear legal authority to collect 
data causes delays as they attempt to gather information for analysis and reporting. 

Limitations of the study 


A key limitation to the study is that the protocol depended upon the cooperation of Chuuk 
State Department of Education representatives and the insights of the data specialists and 
department officials. Documentation of the EMIS (and the education system as a whole) is 
not formally catalogued and is rarely publicly available, thus limiting accessibility and inde- 
pendent review. The study team was limited to the documentation and evidence provided 
in the focus groups by Chuuk data specialists to support their scores for the indicators. 

Allowing only eight data specialists in the focus group session might be considered another 
limitation; however, the data management function in the Chuuk State Department of 
Education is supported by a very small team, and all significant positions related to data 
collection, analysis, and dissemination for the department’s agencies were represented in 
the focus group. 

The study team logged the responses of the data specialists and worked with them to 
identify the supporting documentation referenced during the focus group sessions. As 
a qualitative protocol, however, the assessment of many indicators relied heavily on the 
self-reported perceptions and opinions of Chuuk State Department of Education officials 
and data specialists. The focus group format provided an opportunity for the data special- 
ists to learn each other’s perceptions and viewpoints as well as to review any materials 
their colleagues offered in support of a score, but the effect that peer pressure might have 
had on participant responses is unknown. Given the disparity of responses to some proto- 
col items, this did not seem to be a limitation of the study. 


Data specialists 
may want to 
investigate 
whether the noted 
lack of clear legal 
authority to collect 
data causes delays 
as they attempt to 
gather information 
for analysis and 
reporting 


Another limitation of this study is that the psychometric properties of the SABER EMIS 
Assessment Tool are unknown, although World Bank researchers have indicated posi- 
tive results for consistency across assessors and participants in six island nations in the 
Caribbean (World Bank, 2011). However, the specific inter-rater reliability measures from 
this study are unknown. An initial study by McREL International using the same assess- 
ment framework at the Federated States of Micronesia National Department of Educa- 
tion showed consistency in scores across focus group participants. The tool is designed to 
promote discussions among those engaged in the daily work of operating the state EMIS 
as they consider its value and the quality of various system quality aspects, and then doc- 
ument the various viewpoints and perceptions using quantitative scoring scales. Because 
scores reflect the perspectives of the individuals interviewed, they are not an objective 
view of the EMIS quality. 

A final limitation is that respondents might have offered biased responses because they 
worked with different parts of the Chuuk EMIS and were therefore more familiar with 
some system issues than with others. The focus group format and the use of a transparent 
protocol were designed to mitigate individual biases. However, the focus group format itself 
has the potential for biasing responses because individual confidentiality cannot be guar- 
anteed, especially in a small group. 
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Appendix A. Literature review of education 
management information systems 


Education management information systems (EMISs) are designed to provide compre- 
hensive, integrated, relevant, reliable, unambiguous, and timely data to education leaders, 
decisionmakers, planners, and managers. More simply, an EMIS should provide “the right 
people with the right information at the right time to make best decisions, planning and 
monitoring in the best interest of the organization” (Bhatti & Adnan, 2010, p. 1). The 
quality of any data management system (for example, an EMIS, a state longitudinal data 
system, or a data warehouse) is key to ensuring that education policy, planning, and strat- 
egy decisions are grounded in accurate data (Data Quality Campaign, 2010; Mohamed, 
Kadir, May-Lin, Rahman, & Arshad, 2009; Regional Educational Laboratory Southeast, 
2010; World Bank, 2011). Assessing and benchmarking a data management system can 
also provide policymakers with the opportunity “to gauge. . .how its education system com- 
pares to other systems internationally” (Abdul-Hamid, 2014, p. 14). 

EMISs are seen as a “primary mechanism for monitoring progress and for fostering 
accountability” as education systems move from concern about education coverage to 
concerns about education quality (World Bank, 2011, p. 3). Accurate and reliable infor- 
mation is essential to inform policy planning and policy evaluation (Hua & Herstein, 
2003; World Bank, 2011). For example, in Jamaica, the EMIS helped inform policy briefs 
related to school attendance, teacher qualifications, and test performance (Cassidy, 2006). 
The cost of improving data quality is usually less than the cost of making decisions based 
on bad information (World Bank, 2011). To gauge the quality of an EMIS, it is essential 
to understand whether the system meets key standards for consistency and completeness 
(Mohamed et al, 2009). 

The U.S. Department of the Interior required in 2011 that the Federated States of Micro- 
nesia develop a comprehensive strategy to improve its common centralized data collec- 
tion and analysis (U.S. Department of the Interior, 2010) and reflected this requirement as 
Joint Economic Management Committee (JEMCO) Resolution 2010-6 (Joint Economic 
Management Committee, 2010). This literature review lists previous audits, reviews, and 
assessments of Federated States of Micronesia data systems and discusses the literature on 
key elements of an effective data system and challenges for effectively developing or main- 
taining a data system. The review demonstrates the need for the current study. 

Prior literature related to education management information systems in the Federated States of 
Micronesia 

In the Federated States of Micronesia, improving the quality of data and the rigor of 
education data systems is a central issue (Levine, 2010; U.S. Government Accountabil- 
ity Office, 2006, 2013). The lack of complete, reliable, consistent, and high-quality data 
impedes effective analysis and education reform (U.S. Government Accountability Office, 
2013). The country’s chief state school officers have indicated a need for improvement in 
the timeliness of data reporting from states to the Federated States of Micronesia National 
Department of Education (NDOE) as well as a need to fully implement a comprehensive 
data delivery system (Federated States of Micronesia, 2009). In three annual education 
indicators reports to JEMCO, NDOE articulated challenges to timely data submission and 
data verification (Federated States of Micronesia, 2012, 2013, 2014). The department’s most 
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recent audit, conducted more than five years ago (Wrembeck & Fenlon, 2010), noted that 
no formal data collection or verification policies or procedures existed at the national or 
state levels, suggesting a need to establish minimum standards to strengthen data man- 
agement, collection, and verification. The audit recommended that NDOE “implement a 
standardized data information collection system across all education departments to allow 
for uniformity and accuracy of data collection” (Wrembeck & Fenlon, 2010, p. 11). 

A 2014 study examined NDOE’s EMIS using the System Assessment and Benchmarking 
for Education Results (SABER) EMIS Assessment Tool, which rates six aspects of quality 
using four benchmark score ranges. No aspect of system quality was ranked as the two 
highest benchmark levels (mature and established). Accuracy and reliability of education 
statistics was the highest-rated aspect (at the emerging benchmark level), and methodolog- 
ical soundness was rated lowest (at the latent level; Cicchinelli & Spencer, 2014). A slight- 
ly revised version of the SABER tool was used for the current study and for studies in the 
other three states (Kosrae, Pohnpei, and Yap); findings for those states will be published in 
separate reports. 

Key elements of an effective data system 

Several common elements define an effective and comprehensive data system. The man- 
agement functions of an EMIS include collecting, storing, integrating, processing, organiz- 
ing, and disseminating education data and statistics in a timely and reliable fashion (Hua 
& Flerstein, 2003). Accuracy and consideration of the “relevance of education data and 
education statistics” are also key elements (World Bank, 2011, p. 5). 

The Data Quality Campaign, a nonprofit advocacy organization that provides tools, 
resources, and guidance on data use to policymakers, outlines 10 essential elements of a 
healthy longitudinal data system. These elements include use of unique student identifiers 
that enable data to be linked over time and across different databases, a teacher identifier 
system to link student and teacher data, capacity to link preK-12 and higher education 
student records, and a data audit system to help ensure accuracy and quality (Data Quality 
Campaign, 2013). The 10 essential elements are referenced throughout the literature and 
help to define the framework for a comprehensive data system. 

Integration or centralization of disparate data systems, applications, and information is 
another core element of an EMIS (Cassidy, 2006; LaPointe et al., 2009). Overall, “the 
development and maintenance of an integrated EMIS requires a high degree of coordina- 
tion and collaboration at all levels in the education system, as well as with other ministries 
and with external agencies” (Cassidy, 2006, p. 17). 

Integrity of education statistics is another core element. In the manual for administration 
of the SABER EMIS Assessment Tool, the World Bank identifies integrity as crucial to 
ensuring that the general public can trust education data — and trust that the data are not 
swayed or influenced by political forces (World Bank, 2011). 

The SABER EMIS Assessment Tool prerequisites of quality aspect, which includes both 
the legal and institutional environment and the availability and adequacy of resourc- 
es, articulates the importance of cultivating the knowledge and skills around the use of 
data (Cassidy, 2006). Establishing a clear understanding of who is responsible for various 
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activities of collecting education data is also central (Powell & Trucano, 2006) as are the 
“organizational processes and institutional incentives that drive information use” (Crouch, 
Enache, & Supanc, 2001, p. 49). 

Another key element of an EMIS is the accuracy and reliability of statistical processes (for 
example, data collection and analysis; Powell & Trucano, 2006). Accessibility of education 
data, such as the format for dissemination and a plan for distribution, is also important 
(Powell & Trucano, 2006). Timeliness of data production, which relates to the serviceabih 
ity aspect, is also core to an effective EMIS, particularly because “obsolete data. . .may not 
have much value for use, resulting in missed intervention opportunities” (Hua & Herstein, 
2003, p. 5). 

Challenges around developing or maintaining an effective data system 

Challenges that can arise within an EMIS include version compatibility problems in appli- 
cation software (Mohamed et al., 2009), insufficiently experienced or skilled staff, lack 
of training resources, varying authority over data collection and reporting (McDonald, 
Andal, Brown, & Schneider, 2007), and a lack of communication capacity across data 
systems (Means, Padilla, DeBarger, & Bakia, 2009). 

Version incompatibility poses obstacles for generating usable, effective data that would be 
relevant and applicable for policymakers (Powell & Trucano, 2006). Resource and tech- 
nology constraints (for example, a need for specific software) can also pose a challenge 
(Bandy, Burkhauser, & Metz, 2009; Powell & Trucano, 2006). A typical resource con- 
straint is staff and institutional capacity to effectively employ the data collected (Bandy 
et al, 2009; Cassidy, 2006; LaPointe et al, 2009). Lack of data or system integration into 
a cohesive EMIS limits the ability to establish a functioning monitoring and evaluation 
system, as well as a process for planning and guiding policy (Hua & Herstein, 2003). Man- 
agement may stop relying on data that are late or unreliable, “resulting in ineffective plan- 
ning and budgeting, monitoring and evaluation, policy analysis, and policy-making” (Hua 
& Herstein, 2003, p. 6). 

Some administrators or policymakers may have limited experience using data to guide 
their decisionmaking. Effort is needed to instill a culture of data use at the organizational 
level (Levine, 2010; Powell & Trucano, 2006). An Asian Development Bank report iden- 
tifies this challenge as particularly relevant for at least one state in the Federated States of 
Micronesia (Levine, 2010). 
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Appendix B. Data sources and methodology 


This appendix details the data sources used in the study, the characteristics of focus group 
participants, the data collection methods, and the data processing and analysis used. Each 
of the four benchmark levels for each of the five system aspects of quality are described in 
table Bl. 

Data sources 

Recognizing the need for quality data systems, the International Monetary Fund devel- 
oped the Data Quality Assessment Framework in 2003 to provide stakeholders with a 
flexible structure for the qualitative assessment of statistics in a number of subject areas 
(International Monetary Fund, 2003). The framework identifies quality-related features of 
the governance of statistical systems, statistical processes, and statistical products. 

Building on this framework and applying it to education management information systems 
(EMISs), the World Bank developed the System Assessment and Benchmarking for Edu- 
cation Results (SABER) EMIS Assessment Tool to provide stakeholders with an objective 
scoring tool that allows comparison within a country over time as well as comparison 
across countries. The SABER EMIS Assessment Tool can assess and monitor EMISs on 
aspects of quality (each aspect is described below), which are described by a series of indi- 
cators (each aspect had 7-11 indicators). In addition to correcting typographical errors, 
Regional Educational Laboratory Pacific revised this tool in the following ways. Focusing 
questions were added to each indicator to help clarify the salient points of the scoring 
rubric. One quality aspect, methodological soundness, was removed from the ratings 
because its primary focus on applying international standards was not applicable to com- 
parisons within and across states of the Federated States of Micronesia. Data specialists 
adhere to standards that apply within the Federated States of Micronesia (for example, 
those necessary for compliance with Joint Economic Management Committee regula- 
tions), rather than the United Nations Educational, Scientific and Cultural Organization 
(UNESCO) international standards referenced in the SABER tool. The tool has been 
applied in all four states and at the national level; the resulting data were used as the basis 
for this and other studies. 

Data specialists in the state of Chuuk assigned scores for each indicator in the revised 
SABER EMIS Assessment Tool. The tool provides a focus group protocol (see appendix 
C) with 46 indicators organized into five aspects of quality that characterize EMISs. The 
aspects are: 

• Prerequisites of quality. Assesses whether the institutional frameworks that govern 
the information systems and data reporting and the supporting resources are 
available and adequate. This aspect includes 11 indicators and is divided into two 
subsections. For the first subsection on legal and institutional environment, four 
indicators of quality require that the laws, policies, and institutional arrangements 
clearly support the collection and reporting of education data and statistics. For 
the second subsection on whether human, financial, and computing resources are 
adequate to the task, seven indicators assess the resources, processes, and proce- 
dures to ensure that resources are used effectively and efficiently. 

• Integrity of education statistics. Assesses the professionalism, objectivity, transpar- 
ency, and ethical standards by which data staff operate and by which the data 
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and statistics are reported. This aspect includes nine indicators and is divided into 
three subsections. The first subsection, with four indicators, requires that statistical 
policies and practices are guided by professional principles. The second subsection, 
also with four indicators, requires that statistical policies and practices are trans- 
parent. The last subsection contains one indicator that requires that guidelines for 
staff behavior are in place and well known to the staff. 

• Accuracy and reliability of education statistics. Assesses whether the data being col- 
lected and stored and the statistical techniques used are sound and that statistical 
reports being compiled and reported reflect reality. This quality aspect includes 10 
indicators. 

• Serviceability (relevance, timeliness, and consistency). Assesses whether the data 
being collected and the education statistics being generated are relevant and avail- 
able to inform policy and practice decisions and are consistent over time so prog- 
ress can be measured. This aspect includes seven indicators. 

• Accessibility. Assesses whether education statistics (and their underlying metadata) 
are made available in clear and understandable ways to stakeholders. This quality 
aspect includes nine indicators. 

Each of the 46 indicators has an associated question and scoring rubric, most with descrip- 
tors that identify five score points of .00, .25, .50, .75, and 1.00, from which data specialists 
are asked to select. In three cases (indicators 3.1, 3.2, and 3.5; see appendix C), the score 
depends on how many listed characteristics — short phrases describing key elements of the 
indicator — are present in the data system. 

Characteristics of focus group participants 

The Chuuk state director of education identified eight data specialists whose experience 
and familiarity with the EMIS qualified them to respond to questions regarding the sys- 
tem’s quality and characteristics. These data specialists included a department chief, a 
transition specialist, assessment and statistics specialists, data clerks, and an individual 
working closely with outer island schools. 

Data collection methods 

Two weeks before the interview, these data specialists received the protocol and the 
consent form they would be asked to sign (see appendix C). They were invited to bring any 
physical evidence — regulations, logs, forms, and other artifacts — that supported their view 
on how an indicator should be rated. The focus group interview took place in one day. 
Each indicator and its associated question were read aloud, and data specialists were asked 
to discuss them as a group. Following the discussion, each specialist was asked for a score. 
Unanimity of response was not required for any question. Responses from each specialist 
were logged and projected on an overhead screen so that the data specialists could verify 
their responses. At the end of the session on each aspect, data specialists were asked to 
review the section as a whole and consider whether the scores accurately reflected their 
expressed views. 
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Data processing and analysis 


The scores for each question on the revised SABER focus group protocol were analyzed 
using descriptive statistics, identifying the high, low, and average scores from the data spe- 
cialists. Next, the scores for the indicators under each of the five aspects of quality were 
averaged to identify the appropriate benchmark level for each aspect. Finally, the average 
of all questions was calculated to identify the overall benchmark level for the system. The 
questionnaire is shown in appendix C and the scores for each indicator and aspect of 
quality are given in appendix D. Table B1 describes the four benchmark levels for each of 
the five aspects. 


Table Bl. System Assessment and Benchmarking for Education Results benchmark descriptions 


Benchmark level 

(score range) Benchmark description 


Overall average score 


Latent 

(.00-. 30) 

Lacking statistical infrastructure; little government commitment and use of data; greater needs for 
improving quality of national data than for internationally comparable data. 

Emerging 

(.31-59) 

Basic data channels in place though still weaknesses in reporting by providers; some commitment to 
data use; data are still fragmented across ministries; coverage and relevance needs large improvement; 
some regional benchmarks used. 

Established 

(.60-.79) 

Stable channels of data collection and production; there is a clear strategy and investment in data and 
statistics; more varied sources of data including sample-based surveys; some emerging policy issues 
are addressed in terms of measurement; regional and international comparisons are frequently cited. 

Mature 

(.80-1.00) 

Integrated system of information across state and non-state providers; strong links between users 
and producers of data ensure responsiveness to relevant policy issues and data use; systems and 
information demands are often more complex; data on individuals regularly collected or tracked; 
international comparisons used widely and help to drive policy reforms. 

Prerequisites of quality 


Latent 

(.00-. 30) 

No data sharing or coordination among agencies; low levels of confidentiality in the use of information; 
unclear legal mandate for collecting statistics; little concern for data quality or for the needs of data 

users. 

Emerging 

(.31-59) 

Data sharing but ad-hoc coordination among agencies; confidentiality assured but users not aware of 
it; efficient use of inadequate resources; users' needs are considered sporadically, and data quality 
variations commonly found despite efforts at improving data quality. 

Established 

(.60-.79) 

Data sharing and coordination in place; confidentiality assured; legal framework indirectly in place 
through the legal mandate of the census agency; more efficiency in resource use is needed; users’ 
needs assessed but infrequently; data quality processes in place but enforcement needs improvement. 

Mature 

(.80-1.00) 

Data sharing and coordination in place; confidentiality assured; legal framework in place; efficiency in 
resource use at acceptable levels; users’ needs assessed yearly; data quality processes in place and 


enforced on a regular basis. 

Integrity of education statistics 

Latent Statistics are often modified; professionalism of staff is not promoted; technical decisions are based 


(.00-. 30) 

on statistical and political considerations; statistics are produced with major omissions on institutional 
responsibilities and user considerations. 

Emerging 

(.31-59) 

Statistics are not impartial; professionalism of the staff is promoted on a limited basis; errors in 
statistics are corrected sporadically; statistics have institutional backing but chain of responsibility 
is unclear; major changes in methods, source data and techniques are sporadically made public and 
guidelines for staff behavior are short and incomplete. 

Established 

(.60-79) 

Statistics are impartial; professionalism of the staff is promoted; errors in statistics are corrected 
regularly; statistics have institutional backing but chain of responsibility is unclear; major changes in 
methods, source data and techniques are made public and guidelines for staff behavior are in place, but 
need revisions or improvements. 


(continued) 
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Table Bl. System Assessment and Benchmarking for Education Results benchmark descriptions 

(continued) 


Benchmark level 

(score range) Benchmark description 


Mature 

(.80-1.00) 

Statistics are impartial; professionalism of the staff is promoted; errors in statistics are always 
corrected; statistics’ institutional backing is clear and the chain of responsibility is easily determined; 
major changes in methods, source data, and techniques are always made public and good guidelines for 


staff behavior are in place. 

Accuracy and reliability of education statistics 

Latent Source data are not assessed or assessed only sporadically; the compilation of source data follows 


(.00-. 30) 

methods that do not comply with international standards; intermediate results are not validated with 
data from other sources; statistical discrepancies are investigated and corrected sporadically or on an 

ad-hoc basis. 

Emerging 
(.31-. 59) 

Source data are assessed with some regularity; source data come from comprehensive data collection 
tailored to the country's condition; the compilation of source data follow methods that loosely comply 
with international standards; intermediate results are validated with other data sources only when 
discrepancies are large and easily noted; statistical discrepancies are investigated and corrected 
sporadically or on ad-hoc basis. 

Established 

(.60-.79) 

Source data are assessed regularly; source data come from comprehensive data collection tailored 
to the country’s condition; the compilation of source data follow international standards; intermediate 
results are validated with other data sources if discrepancies are easily noted; statistical discrepancies 
are investigated and corrected regularly. 

Mature 

(.80-1.00) 

Source data are assessed regularly; source data come from comprehensive data collection tailored 
to the country’s condition; the compilation of source data follow international standards; intermediate 
results are always validated with other data sources; statistical discrepancies are always investigated 


and corrected. 

Serviceability (relevance, timeliness, and consistency) 

Latent Statistics are published at irregular intervals and their timeliness is inconsistent with international 


(.00-. 30) 

standards; statistics are inconsistent within the data set and with other data sources; revisions are 
sporadic or absent and preliminary data are not identified. 

Emerging 

(.31-59) 

Statistics are published at regular intervals but their timeliness is inconsistent with international 
standards; statistics are sometimes inconsistent within the data set and with other data sources; 
revisions are sporadic and preliminary data are rarely identified. 

Established 

(.60-.79) 

Statistics are published at regular intervals and their timeliness is consistent with international 
standards; statistics are consistent within the data set and with other data sources; revisions are regular 
but preliminary data are not always identified. 

Mature 

(.80-1.00) 

Statistics are published at regular intervals and their timeliness is consistent with international 
standards; statistics are consistent within the data set and with other data sources; revisions are regular 


and preliminary data are always identified. 

Accessibility to stakeholders 

Latent The presentation of statistics is confusing; dissemination is inadequate and sporadic; documentation 


(.00-. 30) 

is inadequate or insufficient; level of detail does not conform to user needs, and there is no list of 
publications available to users. 

Emerging 

(.31-59) 

The presentation of statistics needs improvement for better interpretation; dissemination is adequate 
but sporadic; documentation is deficient; level of detail conforms to user needs, but there is no list of 
publications available to users. 

Established 

(.60-.79) 

The presentation of statistics is adequate for their interpretation; dissemination is adequate but may 
benefit from including online media; documentation is good; level of detail conforms to user needs, but 
there is no list of publications available to users. 

Mature 

(.80-1.00) 

The presentation of statistics is good for their interpretation; dissemination is good and it includes online 
media; documentation is good and the level of detail conforms to user needs, and there is a catalog of 
publications available to users. 


Note: Latent indicates that the process or action required to improve the aspect of quality is not in place; emerging indicates that the 
process or action is in the process of implementation; established indicates that the process or action is in place and meets standards; 
mature indicates that the process or action is an example of best practice. 

Source: World Bank, 2011, pp. 46-48. 
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Appendix C. Data collection instruments 


This appendix includes the focus group protocol and the consent form provided to each 
participant before the focus group session for this study. The protocol was adapted from a 
World Bank protocol developed for the System Assessment and Benchmarking for Edu- 
cation Results (SABER) Education Management Information System (EMIS) Assessment 
Tool. 

The revised SABER EMIS protocol differs from the World Bank’s protocol in the following 
ways: it includes focusing questions for each indicator, it includes minor clarifying revisions 
(for example, revised introductory text for each aspect; references to “Ministry of Educa- 
tion” replaced with “Department of Education”), and it deletes one of the aspects of quality 
that was not relevant for this study (see appendix B). Finally, two of the scores for indicator 
4.7 were swapped (.50 and .75) to correct an error. All other aspects of the protocol (that is, 
the aspects of quality, indicators, and scoring rubrics), with the exception of typographical 
errors, remain unchanged from the original. 

Protocol for the focus group sessions using the SABER EMIS Assessment Tool aspects of quality, 
indicators, and scoring rubric 

There are five aspects of quality that help define the attributes of an EMIS. For each 
aspect, there are anywhere from 7 to 11 indicators that identify the important elements of 
that category. For each indicator, you will be asked a focusing question and provided with 
clarifying statements and a scoring rubric. 

For example, under the aspect “Accuracy and reliability” a focusing question asks: Are interme- 
diate results always validated against other information where applicable 1 Specifically, you’ll be 
asked which best describes the frequency with which intermediate results are validated in Chuuk: 

1 Intermediate results are always validated against other information where applicable 

.75 Intermediate results are validated most of the time against other information where 
applicable 

.50 Intermediate results are sometimes validated against other information where applicable 

.25 Intermediate results are rarely validated against other information where applicable 

0 Intermediate results are not validated against other information where applicable 

Please provide a score based upon your best understanding of current practices, providing 
any documents, codebooks, or other materials that you believe support your score. 

Aspect 1. Prerequisites of quality 

Aspect 1.1 Legal and institutional environment (subsection) 

The objective of assessing this aspect of quality is to determine the degree to which the 
legal and institutional environments are supportive of educational statistics. 
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Based on the quick analysis of the bulleted items, please assign a score to each of the 
following four indicators using the scoring table attached to each indicator. Indicate your 
rating by circling the appropriate score. 

Indicator 1.1 The responsibility for collecting, processing, and disseminating statistics is 
clearly specified. 

Question: Is the responsibility for collecting, processing, and disseminating statistics clearly 
specified in law that clearly assigns such tasks to an agency or institution; do all working 
arrangements with other institutions reflect this assignment of responsibility? 

• A law exists that assigns the primary responsibility to an institution or an agency 
for the collection, processing, and dissemination of education statistics. The law 
can be in the form of a statistical law or other formal instrument (i.e., executive 
decree). 

• Working arrangements with other institutions are consistent with this assignment 
of responsibility. 


Indicator 1.1 Choose one response in the response matrix and clarify with comments whenever necessary. 


Responsibility 

0 

.25 

.50 

.75 

1 

for collecting and 
disseminating 

education data is 
clearly specified. 

No lines of respon- 
sibility defined and 

no law. 

Limited agreement 
on responsibilities 

and no law. 

Wide agreement on 
responsibilities but 

no law. 

Law exists but 

vague on respon- 
sibilities; it needs 
clarification and/or 
updating. 

Law with clear roles 

and responsibilities 
being implemented. 


Indicator 1.2 Data sharing and coordination among data-producing agencies are adequate 
to facilitate data sharing and cooperation between the education statistics agency and 
other data producing agencies. 

Question: Are the formal data sharing and coordination arrangements between the edu- 
cation statistics agency and other data producing agencies sufficient to ensure the effi- 
cient and timely flow of source data between these agencies, to ensure the consistency of 
methods and results and to ensure sufficient contact, avoiding duplication of effort and 
accounting for the respondent burden? 

• Arrangements or procedures are in place to ensure the efficient and timely flow of 
source data between the education statistics agency and other data producing agencies. 

• Arrangements are in place to ensure the consistency of methods and results. 

• There is regular contact with other data producing agencies to coordinate data require- 
ments, to avoid duplication of effort, and to take into account respondent burden. 


Indicator 1.2 Choose one response in the response matrix and clarify with comments whenever necessary. 


Data sharing and 

0 

.25 

.50 

.75 

1 

coordination among 
different agencies 
are adequate. 

No sharing, no 
arrangements, no 
consistency. 

Informal agreement; 
sporadic/ad hoc 
sharing. 

Informal agreement 

to share exists and 

is mostly imple- 
mented. 

Formal agreement 

to share exists but 

not implemented 
completely. 

There are formal 

arrangements, 
logistics, and 
verification of 

consistency for 

inter-agency 

cooperation. 
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Indicator 1.3 Respondents’ data are always confidential and used only for statistical purpos- 
es. Also, the confidentiality of individual respondents’ data is guaranteed and that guaran- 
tee is widely known. 

Question: Does a law or decree exist that ensures that respondents’ data are always confk 
dential and used only for statistical purposes; does the law or decree and the actual proce- 
dures implemented for enforcement guarantee such confidentiality, and is that guarantee 
widely known? 

This indicator has two subcomponents: the existence of a legal framework for the confiden- 
tiality of individual data, and the existence of actual procedures that ensure confidentiality. 

• The law (or decree) clearly states that whenever school administrative data or 
survey data are collected, the individual responses are confidential and shall only 
be used for statistical purposes. 

• Before answering survey questions, respondents are informed of their obligation 
to provide a truthful response, and the rights to have that response treated in 
complete confidence. 

• There are clearly stated penalties against staff that disclose confidential data and 
those penalties are enforced. 

• Staff reviews all data ready for dissemination for possible indirect disclosure of 
confidential data and devise tables and outputs in a way that prevents disclosure. 

• Access to individual data is restricted to staff who require the information in the 
performance of their duties. 

• Data storage at the education statistics agency is secure enough to prevent unau- 
thorized access to individual data. 

• Confidentiality of data is secure during storage and during the process of the 
destruction of records. 


Indicator 1.3 Choose one response in the response matrix and clarify with comments whenever necessary 


Individuai/personal 

0 

.25 

.50 

.75 

1 

data are kept confi- 
dential and used for 

statistical purposes 
only. 

No law; no confiden- 
tiality. 

Law, but no confi- 
dentiality. 

Law, some confiden- 
tiality. 

Law and confiden- 
tiality, but respon- 
dents not informed 
of their rights. 

Law, confidentiality, 
full rights. 


Indicator 1 A There is a legal mandate that ensures that individuals give their response to 
statistical or survey questions. 

Question: Is there a legal mandate, free of conflicts with other laws or provisions, that 
gives the education data agency the authority to collect and require data from individu- 
als; and in doing so, does the agency provide assistance in order to minimize respondent 
burden and create goodwill to secure respondent cooperation? 

This indicator has two subcomponents: there is a legal mandate that gives the education 
data agency the authority to collect information, and the provisions made by the agency to 
assist individuals in their response to the questions. 

• The agency has the legal authority to collect data required to compile educational 
statistics. 
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• Any conflicts between the legal authority of the agency and other laws or provi- 
sions have been successfully resolved. 

• Individuals know that there are penalties for noncompliance with reporting 
requirements (including misreporting), even if such provisions rarely need to be 
employed. 

• The agency considers carefully the burden for respondents and provides them with 
assistance in completing and submitting forms and information. 

• The agency tries to create goodwill to secure the cooperation of respondents (e.g., 
by answering all the respondents’ questions, by explaining the benefits of the data, 
and by being informative about the overall need for their responses and their use). 


Indicator 1.4 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistical reporting 

0 

.25 

.50 

.75 

1 

is ensured through 
legal mandate and/ 

or measures to en- 
courage response. 

No legal mandate, 

conflicts unre- 
solved, no penal- 
ties, no assistance. 

Informal arrange- 
ments, conflicts 
unresolved, no 
penalties, yes assis- 
tance. 

Legal mandate, con- 
flicts unresolved, 
no penalties, yes 

assistance. 

Legal mandate, 
conflicts resolved, 
no penalties, yes 

assistance. 

Legal mandate, 
conflicts resolved, 
penalties enforced, 
yes assistance. 


Aspect 1.2 Human and material resources are adequate for the task (subsection) 

The objective of assessing this aspect of quality is to determine the degree to which the 
human and material resources are supportive of educational statistics. 

Based on the quick analysis of the bulleted items, please assign a score to each of the fob 
lowing seven indicators using the scoring table attached to each indicator. Indicate your 
rating by circling the appropriate score. 

Indicator 1.5 Staff, financial, and computing resources are commensurate with statistical 
programs of the education data agency. 

Question: Are personnel qualified for their positions, trained to ensure compliance with 
international standards, and retained to ensure that there are sufficient numbers of staff 
adequate to perform the required tasks? Are computer hardware, software, and storage 
maintained and updated to meet the demands of the required tasks? 

The issue of resources is central to data quality. Now that computing power is relatively 
inexpensive, the issue of staff training and staff quality becomes more pressing. As a cor- 
ollary, the budget assigned to education statistics becomes a policy issue, since it should be 
sufficient to pay good staff adequately and provide them with sufficient resources to comply 
with the requirements of high quality data. 

To facilitate rating this indicator it can be useful to examine the following topics: 

Staff resources are adequate to perform required tasks. 

• Overall, the number of the staff is adequate to perform the required tasks. 

• The qualifications of the staff are adequate and they are given continuous on-the' 
job training to comply with international statistical standards. 

• Staff retention is a priority of agency management. 
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Computing resources for compiling statistics are adequate to perform required tasks. 

• Software are continually updated and well adapted to perform existing and emerg- 
ing tasks. 

• Hardware installation is distributed adequately to ensure efficient processing of 
data and management of the databases. 

• Hardware and software security issues are adequate to ensure compliance. 

Financial resources are adequate to perform required tasks. 

• Overall, financial resources are adequate to perform required tasks and commen- 
surate with the overall resources available within the education sector. 

• There is a projection of future budgetary needs derived from an action plan. 


Indicator 1.5 Choose one response in the response matrix and clarify with comments whenever necessary 


Staff, facilities, 

0 

.25 

.50 

.75 

1 

computing resourc- 
es, and financing 

are commensurate 

with the activities. 

Short on staff, short 
on computers, no 
training, no server, 

and outdated soft- 
ware. 

Staff insufficient, 
training required, 

75% of comput- 
ers and software 

and storage need 
updating. 

Staff is suffi- 
cient but training 
required, 50% of 
computers and 
software need up- 
dating, but storage 
is adequate. 

Staff is sufficient, 
training is required, 
25% of computers 
need updating but 

software and stor- 
age are adequate. 

Staff is sufficient, 
good training, 
enough computers 
and storage, 
updated software. 


Indicator 1.6 Processes and procedures are in place to ensure that resources are used 
efficiently. 

Question: Are processes and procedures in place that reflect efficient management of 
human and physical resources and effective monitoring of resource use, and ensure good 
data management, including the careful design of collection instruments and efficient 
compilation of data? 

This indicator is very important because it gives a quick diagnostic of the agency. As a 
result, the bullet points below should help determine your score as well as future courses of 
action. 

• Managers in the education statistics agency promote a policy vision and a direc- 
tion that is shared with the staff. 

• Efficiency is enforced by ensuring consistency in concepts, definitions and meth- 
odologies across the different units and agencies dealing with education statistics. 

• Data collection instruments are carefully designed to avoid duplication of infor- 
mation and lengthy processes in compiling data. 

• Data compilation procedures are managed to minimize processing errors such as 
coding, editing, and tabulation errors. 

• Periodic reviews of working processes are undertaken to ensure that they are 
improved upon. 

• The data producing agency strives to make the best use of newly emerging oppor- 
tunities, such as computing technology for data processing/dissemination, to 
increase the efficiency of resource use. 
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Indicator 1.6 Choose one response in the response matrix and clarify with comments whenever necessary 


Processes and 

0 

.25 

.50 

.75 

1 

procedures are in 
place to ensure that 

resources are used 

efficiently. 

Management disor- 
ganized, untrained, 
and inefficient; data 
management and 
processes highly 

inefficient. 

Management of 
human and physical 

resources is inef- 
ficient; technical 
data processes with 
duplications and 

errors. 

Management of 
human and phys- 
ical resources is 

inefficient; there 
is no monitoring of 

resource use but 
data management 
procedures just 
need improvement. 

Efficient manage- 
ment and moni- 
toring of physical 
resources, but im- 
provements needed 

in human resource 
management. Data 
management proce- 
dures in place. 

Efficient 

management of 
human and physical 
resources, good 
monitoring of 
resource use, and 
data management 
procedures in place. 


Indicator 1.7 Education statistics meet user needs and those needs are monitored 
continuously. 

Question: Are users of statistics consulted on the design of statistics to be produced and 
asked for feedback as part of the review process; does the agency regularly participate in 
regional and international meetings on statistics? 

The issue of accountability is related to this indicator, since to be accountable the edit- 
cation sector has to produce statistical information that is compatible with the informa- 
tion needs of parents and government. As parents and society require better quality in 
education, their information needs also change, such as the inclusion of standardized test 
scores and the reporting of educational expenditures by households. Hence, monitoring 
user needs and producing the corresponding statistics is essential. 

• There is a regular dialogue within the education department between staff responsi- 
ble for statistics and those responsible for policy on statistical information needs, the 
work plan to meet those needs, and the resources needed to meet the new demand. 

• An established process of review takes place periodically to assess whether the 
program meets the needs of users outside of government. 

• The data producing agency regularly participates in international statistical meet- 
ings and seminars organized by international and regional organizations to inform 
about data provision in other countries. 


Indicator 1.7 Choose one response in the response matrix and clarify with comments whenever necessary 


Education statistics 

0 

.25 

.50 

.75 

1 

meet user needs 

and those needs 

are monitored con- 
tinuously. 

No user consulta- 
tion, no user feed- 
back, no interna- 
tional participation. 

Some user consul- 
tation but no feed- 
back, no interna- 
tional participation. 

User consultation, 
some feedback, no 
international partic- 
ipation. 

User consultation, 

some user feed- 
back, some interna- 
tional participation. 

Users are consulted 

in the design of 

statistics to be 

produced, there 
is user feedback; 
participation in 

international 

meetings. 


Indicator 1.8 Processes are in place to focus on data quality, on monitoring the quality of 
the collection, on processing and dissemination of education statistics, and on the inclu- 
sion of data quality in statistical planning. 

Question: Are processes in place and enforced that focus on data quality, on monitoring 
the quality of data collection, on processing and dissemination of education statistics, and 
on the inclusion of data quality in statistical planning? 
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Although it may sound repetitious, data quality improvements depend a great deal on the 
consideration of data quality as an overarching goal of the agency in charge of education 
statistics. If the agency is obsessed with data quality it should reflect that obsession in the 
implementation of processes and procedures that produce quality data. 

• Agency management is sensitive to all aspects of data quality: integrity, method- 
ological soundness, accuracy and reliability, serviceability, and accessibility. 

• The agency has implemented processes or activities that focus on quality (e.g., 
Total Quality Management, ISO 9000, and external audits such as DQAF). 


Indicator 1.8 Choose one response in the response matrix and clarify with comments whenever necessary 


Processes are in 

0 

.25 

.50 

.75 

1 

place to focus on 
quality. 

No quality aware- 
ness in place. 

Management 
promotes ad hoc 
quality improvement 

measures. 

Management clearly 
committed to im- 
proving quality. 

Quality is a main ob- 
jective of operating 
plan. 

Quality procedures 
in place and 
enforced by 
management. 


Indicator 1 .9 Processes are in place to monitor the quality of the collection, processing, and 
dissemination of statistics. 

Question: Are internal and external reviews regularly scheduled, including opportunities 
for user feedback, in order to monitor the quality of the collection, processing, and dissem- 
ination of statistics? 

• Reviews — such as DQAF — are undertaken to identify problems at the various 
stages of collecting, processing, and disseminating data. 

• There is another agency in government that provides guidance on the quality of 
statistics and on strategies for improving data production. 

• Systematic processes exist to obtain feedback from users on data quality issues. 


Indicator 1.9 Choose one response in the response matrix and clarify with comments whenever necessary 


Processes are in 

0 

.25 

.50 

.75 

1 

place to monitor 
the quality of data 

processes. 

No formal reviews; 
no external reviews; 

no user feedback on 
quality. 

Formal reviews 

every 10 years; no 
external reviews; 

user feedback on 

quality every 10 

years. 

Formal reviews 

every 5 years; 

external reviews 
every 10 years; user 
feedback on quality 
every 5 years. 

Formal reviews 

every 3 years; 

external reviews 
every 5 years; user 
feedback on quality 
every 3 years. 

Annual formal 

reviews; external 
reviews every 3 
years; annual user 
feedback on quality. 


Indicator 1.10 Processes are in place to deal with quality considerations in planning the 
statistical program. 

Question: Are processes in place to ensure careful consideration of tradeoffs among 
aspects of quality (for example, between timeliness, completeness, and accuracy/reliability) 
in planning? 

• Agency management knows the tradeoffs among the aspects of quality (for 
example between timeliness, completeness, and accuracy/reliability). 

• The tradeoffs among the aspects of quality are discussed with users and their views 
are taken into consideration. 

• Decisions on the tradeoffs are explicitly included in the data quality improvement 
program. 
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Indicator 1.10 Choose one response in the response matrix and clarify with comments whenever necessary 


Processes are in 

0 

.25 

.50 

.75 

1 

place to deal with 
quality consider- 
ations in planning 

the statistical 

program. 

There is no aware- 
ness of tradeoffs. 

There is awareness 

about tradeoffs but 

no tradeoff analysis 

is conducted. 

Tradeoff analysis 

conducted in ad hoc 

manner. 

Tradeoff analysis 

conducted occasion- 
ally for preserving 
coverage. 

Tradeoff analysis 
conducted regularly 
for preserving 
accuracy and 
reliability. 


Indicator 1.11 Mechanisms exist for addressing new and emerging data requirements. 

Question: Are meetings held with stakeholders and other data users for feedback on the 
existing portfolio, emerging needs, and on current statistical series and reports? 

• Meetings are periodically held with stakeholders and other data users to review 
the existing portfolio of education statistics and reports to identify any emerging 
data requirements. 

• Users’ feedback on the statistical series and statistical reports are encouraged. 


Indicator 1.11 Choose one response in the response matrix and clarify with comments whenever necessary 


Mechanisms exist 

0 

.25 

.50 

.75 

1 

for addressing new 
and emerging data 
requirements. 

No meetings, no 

feedback. 

Meetings with 
stakeholders every 

5 years and no for- 
mal instruments for 

feedback applied. 

Meetings with 
stakeholders every 

5 years and formal 
instruments for 

feedback applied. 

Meetings with 
stakeholders every 

3 years and formal 
instruments for 

feedback applied. 

Annual meeting with 

stakeholders and 

formal instruments 

for feedback 

applied. 


Aspect 2. Integrity of education statistics 

The objective of assessing this aspect of quality is to determine the degree to which the 
integrity of educational statistics is addressed by the data system. Based on the quick 
analysis of the bulleted items, please assign a score to each of the following nine indicators 
using the scoring table attached to each indicator. Indicate your rating by circling the 
appropriate score. 

Aspect 2.1 Statistical policies and practices are guided by professional principles (subsection) 

The term professionalism refers to the ability of statistical staff to exercise their profession 
with technical independence and without outside interference that could result in the 
violation of the public trust in statistics and in the institution. 

Indicator 2.1 Statistics are impartial. Impartiality is assured because the terms and condi- 
tions under which educational statistics are produced guarantee the professional indepen- 
dence of the agency. 

Question: Is there a law or formal provision that protects the professional independence of 
the data producing institution? 

• There is a law or a formal provision that spells out the professional independence 
of the agency by prohibiting interference from others — including other govern- 
ment agencies — in the collection, processing, reporting, and dissemination of edu- 
cation statistics. 
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Professional staff is protected by a code of professional ethics and this code is sup- 
ported by the Department of Education and/or the national statistical agency. 




Indicator 2.1 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistics are 

0 

.25 

.50 

.75 

1 

produced on an 
impartial basis. 

There is no law 
protecting the 
professional 
independence of 
the data producing 

institution. 

There are informal 

mechanisms avail- 
able for protecting 
professionalism 
of data producing 

institution. 

There is a law 
protecting profes- 
sionalism but it is 

not enforced. 

There is a law pro- 
tecting professional- 
ism but is outdated 

and/or enforced 
unevenly. 

A law is in force 
protecting the 
professional 
independence of 
the data producing 

institution. 


Indicator 2.2 Professionalism is actively promoted and supported within the data-producing 
agency. 

Question: Is professionalism actively promoted and supported through recruitment and 
promotion practices and in expectations for publication of research and analysis that 
include an internal review process? 

• Professional competency plays a key role in recruitment and promotion practices. 

• Professionalism is promoted by the publication of methodological papers and by 
encouraging participation in conferences and meetings with other professional 
groups. 

• Research and analysis undertaken by the data-producing agency for publication 
are subject to internal review and other processes to maintain the agency’s reputa- 
tion for professionalism. 


Indicator 2.2 Choose one response in the response matrix and clarify with comments whenever necessary 


Professionalism 

0 

.25 

.50 

.75 

1 

of staff is actively 
promoted. 

Professionalism of 

staff is ignored. 

Professional cre- 
dentials considered 

for recruitment 

and promotion only 
sporadically. 

Professional creden- 
tials are considered 

for recruitment and 

promotion. 

Professional creden- 
tials are considered 

for recruitment and 

promotion and staff 
are encouraged to 
publish. 

Professional creden- 
tials are considered 

for recruitment and 

promotion and staff 
are encouraged to 
publish. There is a 
peer review process 
in place. 


Indicator 2.3 Choices of sources and statistical techniques as well as decisions about dis- 
semination are informed solely by statistical considerations. 

Question: Are choices of source data and statistical techniques based solely on statistical 
considerations and the methods used publicly documented? 

• Choosing source data (e.g., administrative from school census; data from house- 
hold surveys or population census) and statistical techniques (e.g., processing and 
validation techniques) is based solely on statistical considerations. 

• The choice process is driven by technical reasons and the method used is publicly 
documented. 
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Indicator 2.3 Choose one response in the response matrix and clarify with comments whenever necessary 


Choices of data 

0 

.25 

.50 

.75 

1 

sources and sta- 
tistical techniques 
are made solely by 

statistical consider- 
ations. 

Choice of data 

sources are arbi- 
trary and staff do 

not use technical 

criteria. 

Choice of data 

sources are techni- 
cally justified; staff 

can use technical 

criteria, but they are 
not made public. 

Choice of data 

sources are techni- 
cally justified only 
sometimes; staff 
are encouraged to 

enforce technical 

criteria on an ad 

hoc basis and not 

publicly. 

Choice of data 

sources are techni- 
cally justified; staff 
are encouraged to 

enforce technical 

criteria but not 
publicly. 

Choice of data 

sources are 

technically 
justified; staff are 
encouraged to 

enforce technical 
criteria and publish 

those criteria. 


Indicator 2.4 The appropriate statistical entity is entitled to comment on erroneous inter- 
pretation and misuse of statistics. 

Question: Does the agency comment publicly on erroneous interpretations or misuse of 
education statistics by providing explanatory materials and briefings to the public? 

• The agency producing education statistics maintains the public trust by comment- 
ing publicly on erroneous interpretations or misuse of education statistics. 

• The agency seeks to prevent misinterpretation or misuse of education statistics by 
providing explanatory materials and briefings to the public. 


Indicator 2.4 

Agency is entitled 
to comment on 
erroneous interpre 
tation and misuse 
of statistics. 


Choose one response in the response matrix and clarify with comments whenever necessary 


0 

.25 

.50 

.75 

1 

Agency never com- 
ments on errors or 

misinterpretations 
or provides techni- 
cal explanations in 
public. 

Agency comments 
publicly only on 

technical errors but 

not on misinterpre- 
tations and does 
not provide techni- 
cal explanations. 

Agency comments 
only on technical 
errors and provides 
technical explana- 
tions but does not 
act on misinterpre- 
tations. 

Agency com- 
ments publicly on 
technical errors, 
provides technical 
explanations, and 
comments on mis- 
interpretations only 
under pressure. 

Agency com- 
ments publicly on 
technical errors, 
provides technical 
explanations, and 
comments on mis- 
interpretations on a 

routine basis. 


Aspect 2.2 Statistical policies and practices are transparent (subsection) 

Indicator 2.5 The terms and conditions under which statistics are collected, processed, and 
disseminated are available to the public. 

Question: Is information available to the public about the terms and conditions under 
which the statistics are collected, compiled, and disseminated; the confidentiality of indi- 
vidual responses; and the security measures taken? 

• Information is available to the public about the terms and conditions under which 
educational statistics are collected, compiled, and disseminated; the confidential- 
ity of individual responses; and the security measures taken for storing individual 
data. 

• Statistical publications identify where more information about the agency and its 
products can be found. 
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Indicator 2.5 Choose one response in the response matrix and clarify with comments whenever necessary 


Terms and condi- 

0 

.25 

.50 

.75 

1 

tions are available 

to the public. 

Terms and condi- 
tions and additional 

information are not 

released. 

Terms and condi- 
tions and additional 

information are diffi- 
cult to find, although 
they are available 
on request. 

Terms and condi- 
tions and additional 

information are 

difficult to find, 
although they are 

available. 

Terms and condi- 
tions and links to 

additional informa- 
tion are available 

online only. 

Terms and 

conditions are 

clearly available; 

links to additional 

information are 

clear and open in 
print and online. 


Indicator 2.6 Internal governmental access to statistics prior to their release is publicly 
identified. 

Question: Is information openly available regarding access to statistics prior to their 
release, including who has prior access, at what point that access is given, and that the 
agency in charge of education statistics has responsibility for that process? 

Sometimes statistical results are first available internally among government institutions. 
The main reason for these internal releases may be for internal efficiency, where other 
government offices need the information to produce their own statistical series, and some- 
times there are reasons of coordination in the dates for statistical releases. Such internal 
availability is fine as long as data integrity is not compromised. Still, the public must be 
aware of the practice. 

• Access to statistics prior to release is made public in terms of who has access, and 
at what point of the compilation process access is given. 

• The approval process for the publication of education statistics continues to be the 
responsibility of the agency in charge of education statistics. 


Indicator 2.6 Choose one response in the response matrix and clarify with comments whenever necessary 


Public is aware of 

0 

.25 

.50 

.75 

1 

internal govern- 
mental access to 

statistics prior to 

their release. 

No information 
given about internal 
access to prelimi- 
nary data. 

Information on in- 
ternal access given 
upon request. 

Some information 

on internal access 

to preliminary data 
is publicly available. 

All information 

about internal ac- 
cess to preliminary 
data given upon 
request. 

Information about 

internal access to 

preliminary data is 
openly available. 


Indicator 2.7 Products of statistical agencies/units are clearly identified as such. 

Question: Are products of the agency clearly identified as such, and does the agency request 
and secure attribution when its statistics are reproduced or used in joint publications? 

• Data released to the public are clearly identified as a product of the agency in 
charge of education statistics (e.g., by name, logo, and insignia). 

• In the case of joint publications, the part attributable to the agency is identified. 

• The agency requests attribution when its statistics are used or reproduced. 
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Indicator 2.7 Choose one response in the response matrix and clarify with comments whenever necessary 


Products of edu- 

0 

.25 

.50 

.75 

1 

cation statistics 

agency are clearly 

identified. 

There is no attribu- 
tion to any institu- 
tion in the statisti- 
cal publications. 

Attribution given 
only to the Depart- 
ment of Education 

and no requests are 

made for attribution 

from others. 

Attribution is given 
to the Department 

of Education and 

other entities in the 

publication, but no 
requests for attribu- 
tion from others. 

Attribution given 
to the agency and 

to others but no 

requests for attri- 
bution from others 

enforced. 

Statistical unit is 

clearly identified as 
the source of data, 
clearly identifies 
collaborating 
institutions, and 

attribution is 

requested from 

other users. 


Indicator 2.8 Advance notice is given of major changes in methodology, source data, and 
statistical techniques. 

Question: Is advance notice provided to users of agency statistics whenever there are major 
changes in methodology, source data, or statistical techniques? 

• Users of education statistics are made aware in advance of major changes in meth- 
odology, source data, and statistical techniques. 


Indicator 2.8 Choose one response in the response matrix and clarify with comments whenever necessary 


Advance notice 

0 

.25 

.50 

.75 

1 

is given of major 
changes in method- 
ology, source data, 

and statistical 

techniques. 

No notices are given 
on any changes 
in methodology, 
source data, and 

statistical tech- 
niques. 

Agency sends notice 
of major changes in 
methods, sources, 
and techniques only 
upon request. 

Agency sends notice 
of major changes in 
methods, sources, 
and techniques only 

to selected institu- 
tions. 

Agency gives notice 
of major changes in 
methods, sources, 
and techniques 

several months 

after making the 
changes. 

Agency sends notice 
of major changes in 
methods, sources, 
and techniques 

as soon as the 

decision is made. 


Aspect 2.3 Policies and practices in education statistics are guided by ethical standards 
(subsection) 

Ethical standards — in an intuitive form — are those principles that the general public uses 
to differentiate right from wrong. Following this reasoning, the application of ethical starv 
dards to the agency in charge of education statistics implies that the agency follows clear 
standards of good conduct and that those standards are defined for its staff and the general 
public. 

Indicator 2.9 Guidelines for staff behavior are in place and are well known to the staff. 

Question: Does the agency provide staff clear guidelines regarding conflicts of interest and 
ethical behavior? 

• There are clear guidelines outlining correct behavior when the agency or its staff 
is confronted with conflicts of interest. 

• There are clear guidelines for connecting ethical behavior with staff work. Exam- 
pies of this clause can be the use and misuse of statistics, the use of public property 
to conduct private business, or the alteration of statistics in exchange for money. 

• The reputation of the agency and its management is tied to compliance of ethical 
standards. 
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Indicator 2.9 Choose one response in the response matrix and clarify with comments whenever necessary 


Guidelines for staff 

0 

.25 

.50 

.75 

1 

behavior are in 

place and are well 

known to the staff. 

Guidelines for staff 

behavior are non- 
existent. 

Guidelines for staff 

behavior are vague 

and not communi- 
cated to staff. 

Guidelines for 

staff behavior are 

in place but not 

communicated to 

the staff. 

Guidelines for staff 

behavior are in 

place and are well 

known to the staff. 

Guidelines for 

staff behavior 

are in place, are 

well known to the 

staff, and actively 

enforced. 


Aspect 3. Accuracy and reliability of education statistics 

The objective of assessing this aspect of quality is to determine the degree to which source 
data are accurate and reliable. Based on the quick analysis of the bulleted items, please 
assign a score to each of the following 10 indicators using the scoring table attached to 
each indicator. Indicate your rating by circling the appropriate score. 

Indicator 3.1 Source data are obtained from comprehensive data collection programs that 
take into account country-specific conditions. 

Question: Are the source data obtained comprehensive, taking into account the condi- 
tions specific to Chuuk? 

• Statistics describe the structure and normative characteristics of the education 
system, aligning it as much as possible with the ISCED97 standards. 

• Statistics on enrollment and education resources are collected through a regular 
administrative school census program. 

• Administrative school censuses collect information on the structure of education- 
al system, students, teachers, and educational expenditure. 

• Statistics on the demand for education are collected through household surveys 
and population censuses. 

• Statistics on the quality of learning outcomes are collected through assessments of 
student achievement. 

• Statistics on the environment within schools that impact quality of education are 
collected via school surveys. 

The response matrix for this question is somewhat different from the others. The right-hand cell 
lists all the different elements in a comprehensive dataset of the education system. Scoring is 
done by subtraction, that is, if the source data have the five elements listed in the right-hand cell, 
the score is 1. If the source data have only four of the five elements, the score is .75; if the source 
data have only three of the five elements, the score is .5; if the source data have only two of the 
five elements, the score is .25; and if the source data have only one, the score is 0. 


Indicator 3.1 


Choose one response in the response matrix and clarify with 
comments whenever necessary. 


Source data are obtained from 
comprehensive data collection that 
takes into account country-specific 
conditions. 


Source data includes (1) system structure, (2) regular census on 
enrolment, teachers, school and education finances, (3) education 
demand via household surveys, (4) learning outcomes, and (5) 
school characteristics that impact education quality. 
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Indicator 3.2 Source data reasonably approximate the definitions, scope, classifications, val- 
uation, and time of recording required. 

Question: Do source data reasonably approximate the definitions, scope, classifications, 
valuation, and time of recording required? 

This question refers to the compatibility between education statistics to be produced by the 
Department of Education and the data produced by its sources of data. It should be clear 
that the more compatible and consistent the source data are with the statistics required by 
the education system, the lower the probability of error and the higher the probability of 
having education statistics of good quality. 

• Source data are consistent with the definitions, scope, and classifications of educa- 
tion statistics. 

• Source data are consistent with the time of recording, reference periods, and valu- 
ation of education statistics. 

• Data compilers are aware of differences in concepts and definitions used in the 
source data from those required of education statistics. 

The response matrix for this question is somewhat different from the others. The right-hand cell 
lists all the different elements in a comprehensive dataset of the education system. Scoring is 
done by subtraction, that is, if the source data have the [four] elements listed in the right-hand 
cell, the score is 1. If the source data have only [three] of the [four] elements, the score is .75; if 
the source data have only [two] of the [four] elements, the score is .5; if the source data have only 
[one] of the [four] elements, the score is .25; and if the source data have [none], the score is 0. 


Indicator 3.2 


Choose one response in the response matrix and clarify with 
comments whenever necessary. 


Data are reasonably confined to the 
definitions, scope, classifications, 
and time of recording required. 


(1) All source data complies with the standards and scope of 
education statistics data, (2) there are procedures to update and 
standardize source data as needed, (3) data compilers are aware of 
inter-source differences, (4) proper referencing is done for docu- 
menting different source data. 


Indicator 3.3 Source data are timely. 

Question: Are source data agencies compliant with deadline needs of education statistics; 
and are there follow-up procedures for ensuring compliance? 

• Data collection system provides for the timely receipt of source data. 

• Source data providers are aware of the deadlines set for the reporting of education 
statistics. 

• The education statistics agency employs systematic follow-up procedures to ensure 
the timely receipt of source data. 

• Source data from the school census on enrollments and teachers are provided to 
the area responsible for producing statistics no later than 6 months after the end 
of the school year. 

• Source data on educational expenditures are collected from within the Depart- 
ment of Education and other departments and institutions no later than 6 months 
after the end of the school year. 
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Indicator 3.3 Choose one response in the response matrix and clarify with comments whenever necessary 


Source data are 

0 

.25 

.50 

.75 

1 

timely (6 months 
after event). 

Ad hoc or sporadic 
data exchange 

between educa- 
tion statistics 

and source data 

providers. 

Source data agen- 
cies are compliant 

with deadline needs 

of education sta- 
tistics. 

Source data agen- 
cies are compliant 

with deadline needs 

of education statis- 
tics; education data 
are provided more 
than six months 

after the end of the 

school year to other 
source providers. 

Source data agen- 
cies are compliant 

with deadline 

needs of education 

statistics; there are 
follow-up procedures 
for ensuring compli- 
ance; education data 
are provided more 

than six months 

after the end of the 

school year to other 
source providers. 

Source data agen- 
cies are compliant 

with deadline 

needs of education 

statistics; there are 
follow-up proce- 
dures for ensur- 
ing compliance; 

education data are 

provided within six 

months after the 

end of the school 
year to other source 
providers. 


Indicator 3.4 Source data — including censuses, sample surveys, and administrative records 
— are routinely assessed for coverage, sample error, response error, and non-sampling 
error; the results of the assessments are monitored, and corrections to education statistics 
methods are made and published. 

Question: Are source data routinely audited for accuracy, and is information documented 
regarding issues such as sample selection, missing and imputed data, and sampling errors? 

• Administrative and survey data are audited to check the accuracy of source data 
(e.g., inspection of field collections, random post-enumeration checks). 

• Accuracy of data from all sources used to compile statistics is routinely assessed in 
terms of monitored events, population coverage, and the time frames. 

• Information is compiled on coverage, sampling errors (where applicable), non-re- 
sponse errors (e.g., non-response rates for various socioeconomic groups), and the 
percentage of missing and/or imputed data by methods of imputation. 

• For surveys, sampling standard errors of survey estimates are provided in order to 
form confidence intervals for population values, especially when the estimates are 
based on a small sample. 

• For surveys, sample selection is adjusted when sampling errors become large. 

• Relating to administrative data: 

° The use of school registers is promoted and the accuracy of school registers 
is periodically assessed: students dropping out are removed from the register 
or identified as no longer enrolled, students moving or changing schools are 
removed from the register or identified as no longer enrolled, and the register 
includes all students currently enrolled. 


Indicator 3.4 Choose one response in the response matrix and clarify with comments whenever necessary 


Other data sources, 

0 

.25 

.50 

.75 

1 

such as census- 
es, surveys, and 

administrative re- 
cords, are routinely 

assessed. 

Source data are not 

audited; information 
on sampling errors 
and imputed data 

are not documented 

or unavailable. 

Source data are 
rarely audited; 

information on 

sampling errors and 
imputed data are 

not documented or 

unavailable. 

Source data are 
routinely audited; 

information on 

sampling errors and 
imputed data are 
rarely documented 

or shared. 

Source data are 
routinely audited; 

information on 

sampling errors and 
imputed data are 

documented and 

shared. 

Source data are 
routinely audited; 

information on 

sampling errors and 
imputed data are 

documented and 

statistics staff is 

trained to handle 

these issues. 
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Indicator 3.5 Data compilation employs sound statistical techniques to deal with data sources. 

Question: Does the agency employ sound statistical techniques to deal with data sources? 

• Data compilation procedures minimize processing errors such as tabulation errors 
and report generation. 

• The report forms allow for easy completion of the forms and are appropri- 
ate for computer processing. Forms have also been pilot-tested with a sample of 
respondents. 

• Considerations relating to surveys: target population is defined. 

° Sample frames are available for conducting surveys of statistical units (e.g., 
individual, household, and community), minimizing undercoverage and over- 
coverage, and are updated regularly. 

° Scientific random sampling techniques are used, 
o Sample size is appropriate. 

• Considerations relating to administrative data: Enrollment data are collected 
through a school census. 

• A register of all schools exists, and is used to identify responding and nonrespond- 
ing schools. 

o The register covers all schools, with separate identification of public and 
private schools. 

o In expenditure data, intergovernmental transfers (from one department 
to another or from one level of government to another) are netted out and 
counted only at the level where actual expenditure occurs. 

° Institutions and programs for which education expenditure data are reported are 
the same as those for which staff and enrollment data are reported. If this is not 
the case, data are provided separately on number of full-time-equivalent (FTE) 
students and staff in institutions and programs covered by the expenditure data. 


The response matrix for this question is somewhat different from the others. The right-hand cell 
lists all the different elements in a comprehensive dataset of the education system. Scoring is 
done by subtraction, that is, if the source data have the five elements listed in the right-hand cell, 
the score is 1. If the source data have only four of the five elements, the score is .75; if the source 
data have only three of the five elements, the score is .5; if the source data have only two of the 
five elements, the score is .25; and if the source data have only one, the score is 07 


Indicator 3.5 


Choose one response in the response matrix and clarify with 
comments whenever necessary. 


Data compilation employs sound 
statistical techniques to deal with 
data sources. 


For survey data: (1) random sampling, (2) appropriate sample size. 
For census data: (3) updated registry of all schools (public, private) 
exists to identify responding and non-responding schools. 


Indicator 3.6 Other statistical procedures employ sound statistical techniques. 

Question: Does the agency document data adjustments and transformations and employ 
sound statistical methods in data transformation? 

• Imputation methods, estimation techniques (e.g., sampling weights, calibration 
weights), employ sound statistical techniques. 


* Given the inconsistency of the protocol instructions with the number of elements listed, data specialists 
were instructed to use a revised version of the scoring approach for this indicator. 


C-16 


• Problems regarding non-responses, recall errors, reporting errors, respondents 
effects, interviewer effects, and inappropriate instrument design are addressed. 

• Imputation and estimation methods are appropriate for dealing with missing data 
from administrative records, household surveys and population censuses, sample 
survey or schools, and assessments of student achievement. Proper imputation 
methods are used wherever feasible to handle missing, invalid or inconsistent 
responses. If there is a sizeable part of the population that is not covered by sources 
used for regular compilation of statistics, undercoverage adjustments are made, or 
if such adjustments are not feasible in terms of being statistically defensible, the 
limitation in the coverage of the statistics is described. 

• Where compensation for missing data is not feasible (e.g., if data is not collected 
from private schools), the nature of the missing data is described. 


Indicator 3.6 Choose one response in the response matrix and clarify with comments whenever necessary 


Other statistical 

0 

.25 

.50 

.75 

1 

procedures (data 
editing, transforma- 
tions, and analysis) 
employ sound sta- 
tistical techniques. 

No data adjust- 
ments made when 

needed. 

Some data adjust- 
ments and transfor- 
mations made but 

not documented. 

Data adjustments 

and transformations 

made but not docu- 
mented; statistical 

methods used in 

data transformation 

not to international 

standards. 

Data adjustments 

and transforma- 
tions made but not 

documented; sound 

statistical methods 

used in data trans- 
formation. 

Data adjustments 

and transformations 

documented; sound 

statistical methods 

used in data trans- 
formation. 


Indicator 3.7 Assessment and validation of intermediate data and statistical outputs: Inter- 
mediate results are validated against other information where applicable. 

Question: Are intermediate results always validated against other information where 
applicable? 

• Data from different sources but measuring the same or closely related phenomena 
are compared against each other. 


Indicator 3.7 Choose one response in the response matrix and clarify with comments whenever necessary 


Intermediate results 

0 

.25 

.50 

.75 

1 

are validated 
against other 

information where 

applicable. 

Intermediate results 

are not validated 

against other 
information where 

applicable. 

Intermediate results 

are rarely validat- 
ed against other 
information where 

applicable. 

Intermediate results 

are sometimes vali- 
dated against other 
information where 

applicable. 

Intermediate results 

are validated most 

of the time against 
other information 

where applicable. 

Intermediate results 

are always validated 
against other 
information where 

applicable. 


Indicator 3.8 Statistical discrepancies in intermediate data are assessed and investigated. 

Question: Are statistical discrepancies in intermediate data always assessed and 
investigated? 

• Post-survey data analysis is conducted to monitor statistical discrepancies. 

• Provision is made for immediate follow-up to reconcile data inconsistencies. 
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Indicator 3.8 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistical dis- 

0 

.25 

.50 

.75 

1 

crepancies in 

intermediate data 

are assessed and 

investigated. 

Statistical discrep- 
ancies in interme- 
diate data are not 

assessed and inves- 
tigated. 

Statistical discrep- 
ancies in intermedi- 
ate data are rarely 

assessed and inves- 
tigated. 

Statistical discrep- 
ancies in inter- 
mediate data are 

assessed and inves- 
tigated sometimes. 

Statistical dis- 
crepancies in 

intermediate data 

are assessed and 

investigated most of 

the time. 

Statistical discrep- 
ancies in intermedi- 
ate data are always 

assessed and inves- 
tigated. 


Indicator 3.9 Statistical discrepancies and other potential indicators of problems in statisti- 
cal outputs are investigated. 

Question: Are systematic processes in place (such as checks of demographic data and data 
from previous years) for monitoring errors and omissions? 

• Systematic processes are in place to monitor errors and omissions and address data 
problems. 

• Results are checked against demographic data and other survey/census results. 

• Data are compared with data from earlier years to examine reasonableness of year- 
to-year changes and trends. 


Indicator 3.9 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistical discrep- 

0 

.25 

.50 

.75 

1 

ancies and other 

potential indicators 
or problems in sta- 
tistical outputs are 
investigated. 

There are no sys- 
tematic processes 
(check demographic 
data; check previ- 
ous years) in place 
for monitoring errors 

and omissions. 

There are systemat- 
ic processes (check 
demographic data; 
check previous 
years) in place for 
monitoring errors 

and omissions but 

they are rarely used. 

There are systemat- 
ic processes (check 
demographic data; 
check previous 
years) in place for 
monitoring errors 

and omissions but 

they are not applied 
consistently. 

There are systemat- 
ic processes (check 
demographic data; 
check previous 
years) in place for 
monitoring errors 

and omissions but 

results are not 

made public. 

There are systemat- 
ic processes (check 
demographic data; 
check previous 
years) in place for 
monitoring errors 

and omissions and 

the results are 

made public. 


Indicator 3.10 Studies and analyses of revisions are carried out routinely and used to inform 
statistical processes. 

Question: Are studies and analyses of revisions regularly assessed and analysis of prelimi- 
nary versus revised data conducted in order to assess reliability of data? Are revision find- 
ings made public? 

• Revisions to methodology are assessed regularly. 

• Analysis of preliminary versus revised data is conducted for major data series to 
assess the reliability of the preliminary data and findings are taken into account. 

• Revision findings are made accessible to the data users and compilers. 


Indicator 3.10 Choose one response in the response matrix and clarify with comments whenever necessary 


Studies and analy- 

0 

.25 

.50 

.75 

1 

ses of revisions are 
carried out routinely 
and used internally 
to inform the pro- 
cesses. 

Revisions to meth- 
odology are rarely or 

never made. 

Methods are re- 
viewed; no assess- 
ments of preliminary 
vs. revised data are 

made. 

Methods are re- 
viewed; preliminary 

vs. revised data are 

assessed; no feed- 
back loop imple- 
mented; findings are 
not made public. 

Methods are re- 
viewed; preliminary 

vs. revised data are 

assessed; feedback 
loop implemented; 
findings are not 
made public. 

Methods are re- 
viewed; preliminary 

vs. revised data are 

assessed; feedback 
loop implemented; 
findings are made 
public. 
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Aspect 4. Serviceability (relevance, timeliness, and consistency) 


The objective of assessing this aspect of quality is to determine the relevance, timeliness, 
and consistency of education statistics, as well as the revision policies associated with these 
issues. Based on the quick analysis of the bulleted items, please assign a score to each of 
the following seven indicators using the scoring table attached to each indicator. Indicate 
your rating by circling the appropriate score. 

Indicator 4-1 Periodicity and timeliness: Periodicity follows dissemination standards. 

Question: Are education statistics disseminated annually or at periods that follow interna- 
tionally accepted good practices, as appropriate? 

• Education statistics derived from the administrative school census are disseminat- 
ed annually. 

• The periodicity of other education statistics follows internationally accepted good 
practices. 


Indicator 4.1 Choose one response in the response matrix and clarify with comments whenever necessary 


Periodicity follows 

0 

.25 

.50 

.75 

1 

dissemination stan- 
dards. 

Censuses of en- 
rollment, teachers, 

schools and finan- 
cial data are only 
produced every 5 or 
more years. 

Censuses of en- 
rollment, teach- 
ers, schools, and 

financial data are 

produced every 2-5 
years. 

Censuses of en- 
rollment, teach- 
ers, schools, and 

financial data are 

produced every 2 
years. 

Census of enroll- 
ment is annual but 

census of teachers, 
schools, and financ- 
es are not produced 
annually. 

Censuses of en- 
rollment, teach- 
ers, schools, and 

financial data are 

produced annually. 


Indicator 42 The timeliness of statistics follows internationally accepted good practices. 

Question: Are administrative school census data and other education statistics disseminat- 
ed in accordance with internationally accepted good practices? 

• Statistics derived from the administrative school census are disseminated within 
6-12 months after the beginning of school year. 

• The timeliness of other education statistics follows internationally accepted good 
practices. 


Indicator 4.2 Choose one response in the response matrix and clarify with comments whenever necessary 


Timeliness follows 

0 

.25 

.50 

.75 

1 

international 

dissemination stan- 
dards. 

Administrative 

school census data 

are available 6-12 

months after the 

end of the school 

year. 

Administrative 

school census data 

are available 0-6 

months after the 

end of the school 

year. 

Administrative 

school census data 

are available 6-12 

months after the 

initiation of the 
school year. 

Administrative 

school census data 

are available 2-6 

months after the 

initiation of the 
school year. 

Administrative 

school census 

data are available 

2 months after the 

initiation of the 
school year. 


Indicator 4-3 Consistency: Statistics are consistent within the dataset. 

Question: Are all enrollment and administrative data regularly checked for consistency 
and crosschecked? 
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Consistency is measured in a simple way: The total reported should also be obtained by 
adding the components of such total. 

• Accounting identities between aggregates and their components are observed for 
all involved data. 

• Accounting identities between enrollments, repeaters, dropouts, and demographic 
data are observed. 

• Statistics are crosschecked within the survey, across geographic areas and sub- 
groups of population. 


Indicator 4.3 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistics are con- 

0 

.25 

.50 

.75 

1 

sistent within the 

dataset. 

No consistency or 
crosschecking done 

on the data. 

Consistency check- 
ing done only for 

enrollment data and 

there is no cross- 
checking. 

Consistency check- 
ing done only for 

enrollment data and 

crosschecking done 
regularly. 

Consistency 
checking done only 

for administrative 

census data and 

crosschecking done 
regularly. 

Consistency check- 
ing done for all data 
and crosschecking 
done regularly. 


Indicator 4-4 Statistics are consistent or reconcilable over a reasonable period of time. 

Question: Are consistent time series available for at least five years, and historical series 
reconstructed as possible when changes require? Are revisions and revision methods made 
public, and inconsistencies explained? 

• Consistent time series data are available for an adequate period of time (at least 
five years). 

• When changes in source data, methodology, and statistical techniques are intro- 
duced, historical series are reconstructed as far back as reasonably possible. 

• Detailed methodological notes identify and explain the main breaks and discon- 
tinuities in time series, their causes, as well as adjustments made to maintain con- 
sistency over time. 

• Any unusual changes in economic and demographic trends are explained in the 
analytical text included in the publication and in the database accessible to users. 


Indicator 4.4 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistics are 

0 

.25 

.50 

.75 

1 

consistent or 

reconcilable over a 

reasonable period 

of time. 

Time series are 

available for less 

than 5 years; there 
are no procedures 

for revision of time 

series. 

Time series are 

available for less 

than 5 years; there 
are procedures for 

revision of time 

series; the revision 

methods are not 

public; and incon- 
sistencies are not 

explained. 

Time series are 

available for more 

than 5-10 years; 
there are proce- 
dures for revision 

of time series; the 

revision methods 
are not public; and 

inconsistencies are 

not explained. 

Time series are 

available for 5-10 

years; there are pro- 
cedures for revision 

of time series; the 

revision methods 

are public; and 
inconsistencies are 

explained. 

Time series are 

available for more 

than 10 years; there 
are procedures for 

revision of time 

series; the revision 
methods are public; 

and inconsistencies 

are explained. 
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Indicator 4-5 Statistics are consistent or reconcilable with those obtained through other 
data sources and/or statistical frameworks. 

Question: Are education statistics reconcilable with administrative data, census data, and 
sociodemographic data from other sources? 

• Education statistics are reasonably reconciled with administrative data, census 
data, and socio-demographic data from other sources. 


Indicator 4.5 

Statistics are con- 
sistent or recon- 
cilable with those 
obtained through 
other data sources 
and/or statistical 
frameworks. 


Choose one response in the response matrix and clarify with comments whenever necessary 


0 

.25 

.50 

.75 

1 

Percent difference 

in primary and 
secondary education 

enrollment between 

school-reported fig- 
ures and data from 

household surveys 
is larger than 30 per- 
centage points. 

Percent difference 

in primary and 
secondary education 

enrollment between 

school-reported fig- 
ures and data from 

household surveys is 
between 21-30 per- 
centage points. 

Percent difference 

in primary and 
secondary education 

enrollment between 

school-reported fig- 
ures and data from 

household surveys is 
between 11-20 per- 
centage points. 

Percent difference 

in primary and 
secondary education 

enrollment between 

school-reported fig- 
ures and data from 

household surveys is 
between 5-10 per- 
centage points. 

Percent difference 

in primary and sec- 
ondary education 

enrollment between 

school-reported fig- 
ures and data from 

household surveys 
is lower than 5 per- 
centage points. 


Indicator 4-6 Revision policy and practice: Revisions follow a regular, well established, and 
transparent schedule. 

Question: Are revisions of provisional estimates, methods, and outputs documented and 
made accessible to users, following an established schedule? 

• Adequate documentation of revisions is included in the publication of the statisti- 
cal series and in the database accessible to users. 


Indicator 4.6 Choose one response in the response matrix and clarify with comments whenever necessary 


Revisions follow a 

0 

.25 

.50 

.75 

1 

regular and trans- 
parent schedule. 

There are no revi- 
sions. 

There are ad hoc 
partial formal 
revisions of provi- 
sional estimates, 
methods, and 
outputs. Documen- 
tation available to a 
restricted group. 

There are annual 
partial formal re- 
visions of provi- 
sional estimates, 
methods, and 
outputs. Documen- 
tation available to a 
restricted group. 

There are docu- 
mented formal revi- 
sions of provisional 
estimates, meth- 
ods, and outputs 
every two years. 

There are docu- 
mented annual 

formal revisions 

of provisional esti- 
mates, methods, 
and outputs. 


Indicator 47 Preliminary data and/or revised data are clearly identified and revisions are 
made public. 

Question: Are preliminary and/or revised data clearly identified in public documents? 

• Users are alerted that initially published data are preliminary and subject to 
revision. 

• The revised data are disseminated with the same level of detail as previously pub- 
lished for the data being revised. 

• Revisions to methodology are assessed and explained in the publication of the 
statistical series and in the database accessible to users. 

• Analysis of preliminary versus revised data is published for major data series to 
allow assessment of the reliability of the preliminary data. 
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Indicator 4.7 Choose one response in the response matrix and clarify with comments whenever necessary 


Preliminary and/ 

0 

.25 

.50 

.75 

1 

or revised data are 

clearly identified. 

No preliminary data 
are produced. 

Preliminary and/or 

revised data are not 

identified. 

Preliminary and/ 

or revised data are 

clearly identified but 
not made public. 

Preliminary and/ 

or revised data are 

clearly identified 
but only a portion is 
made public. 

Preliminary and/ 

or revised data are 

clearly identified in 
public documents. 


Aspect 5. Accessibility 

The objective of assessing this aspect of quality is to determine the extent to which edu- 
cation statistics are presented in a clear and understandable manner, whether forms of dis- 
semination are adequate, and whether statistics are made available on an impartial basis. 
Based on the quick analysis of the bulleted items, please assign a score to each of the 
following nine indicators using the scoring table attached to each indicator. Indicate your 
rating by circling the appropriate score. 

Indicator 5.1 Statistics are presented in a way that facilitates proper interpretation and 
meaningful comparisons (layout and clarity of text, tables, and charts). 

Question: Are statistics presented in a way that facilitates proper interpretation and com- 
parisons through effective use of layout, tables, charts, and clarity of text, and are data 
accessible for disaggregation? 

• Education data are published in a clear manner; charts and tables are disseminat- 
ed with the data to facilitate the analysis. 

• Education data offer adequate details and time series. 

• Analysis of current period estimates is available. 

• Data are presented for different degrees of aggregation (e.g., school, region) and 
sub-components (e.g., by gender, by level of education, by age, private and public, 
full-time and part-time), and additional data (e.g., demographic, socioeconomic, 
geographic information) are included. 


Indicator 5.1 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistics are pre- 

0 

.25 

.50 

.75 

1 

sented to facilitate 

proper interpreta- 
tion and compari- 
sons (layout, clarity 
of texts, tables, and 
charts). 

No presentation of 
data outputs. 

Data are not pre- 
sented clearly. 

Clear presentation 
of data; charts have 
no underlying data 
available; disaggre- 
gation of data is not 
presented. 

Clear presentation 
of data; charts have 
underlying data 
available; disaggre- 
gation of data is not 
presented. 

Clear presentation 
of data; charts have 
underlying data 
available; disaggre- 
gation of data is 
possible. 


Indicator 5.2 Dissemination media and formats are adequate. 

Question: Are data released in media and formats that are accessible? 

• Data are first released via an information release, which is then followed by the 
release of a more comprehensive publication. 

• Recently released data and longer time series can be accessed through an electron- 
ic database maintained by the agency producing education statistics. 

• Annual education statistical yearbook can be made available and disseminated. 
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Indicator 5.2 

Choose one response in the response matrix and clarify with comments whenever necessary 

Dissemination me- 

0 

.25 

.50 

.75 

1 

dia and format are 

adequate. 

During the last 5 
years, data were not 

available electroni- 
cally and there is no 
yearbook ready for 

dissemination. 

Data are not 

available 

electronically, 

but there is a 

yearbook ready for 

dissemination. 

During the last year, 

data were available 

electronically 

and there was a 

yearbook ready for 

dissemination. 

During the last 

2-4 years, data 

were available 

electronically 

and there was a 

yearbook ready for 
dissemination. 

During the last 

5 years, data 

were available 

electronically 

and there was a 

yearbook ready for 
dissemination. 

Indicator 5.3 Statistics are released on a pre-announced schedule. 



Question: Are statistics released according to a pre-announced schedule? 

• Education statistics are released according to a pre-announced schedule. 


Indicator 5.3 

Choose one response in the response matrix and clarify with comments whenever necessary 

Statistics are 

0 

.25 

.50 

.75 

1 

released on a 

pre-announced 

schedule. 

Data are not re- 
leased. 

There is no 

pre-announced 

schedule for data 

release. 

There is a pre- 
announced 

schedule for data 

release and the 

data are released 

>6 months later. 

There is a pre- 
announced 

schedule for data 

release and the 

data are released 

0-6 months later. 

There is a pre- 
announced 

schedule for data 

release and the 

data are released 
accordingly. 

Indicator 5.4 Statistics are made available to all users at the same time. 



Question: Are statistics released at the same time to all users? 

• Education statistics are released simultaneously to all users on the date and/or time 
specified in the pre-announced schedule. 

• If the press is briefed in advance, measures are taken to avoid release to the public 
in advance of the regular schedule. 


Indicator 5.4 

Choose one response in the response matrix and clarify with comments whenever necessary 

Statistics are made 

0 

.25 

.50 

.75 

1 

available to all 

users at the same 

time. 

No data are re- 
leased. 

Some of the data 

are released to 

restricted users. 

Most of the time, 
part of the data are 

released to all users 
simultaneously. 

Most of the time, all 

of the data are re- 
leased to all users 
simultaneously. 

All data are re- 
leased at the same 

time to all users. 


Indicator 5.5 Statistics not routinely disseminated are made available upon request. 

Question: Are there procedures in place so that statistics that are not routinely disseminat- 
ed (but non-confidential) can be made available upon request? 

• Non-published (but non-confidential) specialized tabulations (e.g., sub-aggregates 
of units of analysis) are made available upon request. 

• Non-confidential micro-data files (e.g., with information permitting the identifka- 
tion of individual respondents removed) are available to permit analytical use by 
researchers and other users. 

• The availability of non-published statistics and data, and the terms and conditions 
on which they are made available, are publicized. 
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Indicator 5.5 Choose one response in the response matrix and clarify with comments whenever necessary 


Statistics not rou- 

0 

.25 

.50 

.75 

1 

tinely disseminated 

are made available 

upon request. 

Release of non- 

published data 
may compromise 
confidentiality. 

Release of non- 

published data and 

non-confidential 

data is without 

controls. 

Release of 

non-published 

data and non- 

confidential data is 

discretionary. 

There are proce- 
dures in place for 
releasing non- 
published data and 

non-confidential 

data to a restricted 

group. 

There are proce- 
dures in place for 
releasing non- 
published data and 

non-confidential 

data. 


Indicator 5.6 Metadata accessibility: All metadata documentation is available, and differ- 
ences from international standards are annotated. 

Question: Is all metadata documentation (including information on concepts, defini- 
tions, classification and other methodology, data sources, and statistical techniques) made 
available to the public, and are deviations from internationally accepted standards well 
documented? 

• Metadata, including information on concepts, definitions, classification and other 
methodology, data sources, and statistical techniques, are prepared and dissemi- 
nated to the public. 

• Deviations from internationally accepted standards, guidelines, or good practices 
are well documented in the metadata. 

• The metadata is disseminated in a manner that facilitates its access (e.g., websites, 
statistical publications) and its availability is well publicized. 

• Instances where statistical information for the subject area in question makes use 
of data pertaining to other subject areas and produced by other data producers are 
noted, and references are given to descriptions of their methodology and quality. 


Indicator 5.6 Choose one response in the response matrix and clarify with comments whenever necessary 


Documentation on 

0 

.25 

.50 

.75 

1 

concepts, scope, 
classifications, 
basis of recording, 
data sources, and 

statistical tech- 
niques is available, 

and differences 

from internationally 
accepted stan- 
dards, guidelines, or 
good practices are 
annotated. 

No metadata are 

available. 

Metadata, including 
information on con- 
cepts, definitions, 
classifications, 
sources, methodol- 
ogy, and statistical 
techniques are 
incomplete and 

outdated. 

Metadata, including 
information on con- 
cepts, definitions, 
classifications, 
sources, methodol- 
ogy, and statistical 
techniques are 
documented but 

outdated, and avail- 
able upon request. 

Metadata, including 
information on con- 
cepts, definitions, 
classifications, 
sources, methodol- 
ogy, and statistical 
techniques are doc- 
umented, updated, 
and available upon 
request. 

Metadata, including 
information on con- 
cepts, definitions, 
classifications, 
sources, methodol- 
ogy, and statistical 
techniques are 
documented, updat- 
ed, and available to 
public. 


Indicator 5.7 Levels of detail are adapted to the needs of the intended audience. 

Question: Is a regularly updated data catalog available so that users can request details of 
data according to their needs? 

• A brochure has been prepared to inform general users about the statistical series. 

• A comprehensive sources and methods document is produced and updated regular- 
ly to inform analysts and other users of statistics about how statistics are compiled. 
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Indicator 5.7 Choose one response in the response matrix and clarify with comments whenever necessary 


Levels of detail 

0 

.25 

.50 

.75 

1 

are adapted to the 

needs of the intend- 
ed users. 

No data catalog is 
produced. 

Data catalog is 

available to select- 
ed users. 

Data catalog is 

available so users 

can request detail 
of data according to 
their needs. Catalog 
is not updated an- 
nually but selected 

users have access 

to data. 

Data catalog is 

available so users 

can request details 
of data according to 
their needs. Catalog 
is updated annually 
but just selected 

users have access 

to data. 

Data catalog is 

available so users 

can request details 
of data according to 
their needs. Catalog 
is updated annually 
and data are acces- 
sible to users. 


Indicator 5.8 Assistance to users: Contact points for each subject field are publicized. 

Question: Do all statistical releases identify a contact person in case of required assistance, 
are data manuals and/or brochures produced to educate users, and is assistance to users 
monitored through periodic surveys? 

• Prompt and knowledgeable service and support are available to users of statistics. 
All statistical releases identify specific individuals who may be contacted by mail, 
telephone, facsimile, or email. 

• Documentation has been developed (e.g., brochures) to educate users of related 
datasets. 

• Assistance to users is monitored through periodic surveys of users. 


Indicator 5.8 Choose one response in the response matrix and clarify with comments whenever necessary 


Contact points for 

0 

.25 

.50 

.75 

1 

each subject field 
are publicized. 

Statistical releas- 
es do not identify 
contact person. 

Most statistical 

releases identify 
contact person in 
case of required 

assistance. No 

data manuals 

and/or brochures 
are produced to 

educate users and 

assistance to users 

is not monitored. 

All statistical releas- 
es identify contact 
person in case of 
required assis- 
tance. For limited 

and hard to obtain 

data, manuals 
and/or brochures 
are produced to 
educate users and 

assistance to users 

is not monitored. 

All statistical releas- 
es identify contact 
person in case of 
required assistance. 

Data manuals 

and/or brochures 
are produced to 

educate users and 

assistance to users 

is not monitored. 

All statistical releas- 
es identify contact 
person in case of 
required assistance. 

Data manuals 

and/or brochures 
are produced to 

educate users and 

assistance to users 
is monitored though 
periodic surveys. 


Indicator 5.9 Catalogues of publications, documents, and other services, including informa- 
tion on any charges, are widely available. 

Question: Are catalogs of publications and services widely available and updated yearly, 
and do they include information about charges for placing orders and ordering assistance 
that is available? 

• A catalog of publications, documents, and other services to users is available and 
updated each year. 

• The prices of the statistical products and services are clearly disclosed and assis- 
tance is provided in placing orders. 
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Indicator 5.9 Choose one response in the response matrix and clarify with comments whenever necessary 


Catalogues of pub- 

0 

.25 

.50 

.75 

1 

lications and other 

services, including 
information on any 
charges, are widely 
available. 

Catalogues of publi- 
cations and service 

are not available. 

Catalogues of 
publications and 

services are avail- 
able but not updat- 
ed yearly. Prices of 
statistical products 
and services are not 

clearly disclosed. 

Catalogues of 
publications and 

services are avail- 
able and updated 
yearly. Prices of 
statistical products 

and services are not 

clearly disclosed. 

Catalogues of 
publications and 

services are avail- 
able and updated 
yearly. Prices of 
statistical products 
and services are 

clearly disclosed 

but assistance for 

placing orders is not 

available. 

Catalogues of 
publications and 

services are avail- 
able and updated 
yearly. Prices of 
statistical products 

and services are 

clearly disclosed 

and assistance for 

placing orders is 

available. 


Aspect on methodological soundness removed 

The original SABER EMIS protocol includes one aspect that focuses on the UNESCO 
Institute for Statistics (UIS) international standards. Because data specialists within the 
Federated States of Micronesia are required to apply standards set by the NDOE and agen- 
cies and organizations that oversee the distribution of funds to the states and the NDOE, 
such as JEMCO, this aspect is of limited application. The SABER tool is intended in part 
to provide a means to compare countries to one another; however, the purpose of this 
study is to provide states and the NDOE with information that allows comparison within 
the Federated States of Micronesia. Thus, whether international standards were applied 
provides little actionable information. 

Methodological soundness 

The objective of assessing this aspect of quality is to determine the degree to which the 
methodological basis for educational statistics follows internationally accepted standards, 
guidelines, and good practices. Based on the quick analysis of the bulleted items, please 
assign a score to each of the following three indicators using the scoring table attached to 
each indicator. Indicate your rating by circling the appropriate score. 

Indicator 1 Concepts and definitions are in accord with standard statistical frameworks. 

Question: Do the concepts and definitions used by the agency follow regionally and inter- 
nationally accepted standards, guidelines, and good practices? 

• The concepts and definitions follow methodologies recommended by UIS. 


Indicator 1 Choose one response in the response matrix and clarify with comments whenever necessary 


Overall structure, 

0 

.25 

.50 

.75 

1 

concepts and 

definitions follow 

regionally and inter- 
nationally accepted 
standards, guide- 
lines, and good 
practices. 

Structure, concepts, 

and definitions are 

inconsistent from 

year to year, without 
proper documen- 
tation, and without 
consistency with 
regional or interna- 
tional standards. 

Structure, concepts, 

and definitions do 

not have proper doc- 
umentation and may 
or may not be con- 
sistent with regional 

and international 

standards. 

Structure, concepts, 

and some defini- 
tions have proper 

documentation and 

may or may not 

be consistent with 

regional and interna- 
tional standards. 

Structure, concepts, 

and definitions have 

proper documenta- 
tion but definitions 

do not conform with 
regional and interna- 
tional standards. 

Overall structure, 
concepts, and 

definitions follow 

regionally and inter- 
nationally accepted 
standards, guide- 
lines, and good 
practices. 
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Indicator 2 Scope of education statistics are in accordance with international standards, 
guidelines, or good practices. 

Question: Is the scope and conceptual development of education statistics sufficiently 
comprehensive in accordance with international standards, guidelines, and good practices? 

• Education statistics are sufficiently comprehensive in scope and conceptual deveh 
opment to adequately describe the full performance of the education sector. 

• Scope of statistics is adequate in terms of other relevant variables for analytical 
purposes. 


Indicator 2 Choose one response in the response matrix and clarify with comments whenever necessary 


Scope is in accor- 

0 

.25 

.50 

.75 

1 

dance with interna- 
tional standards, 
guidelines, or good 
practices. 

Scope of agency 

statistics covers 

less than 50% of 

UIS indicators. 

Scope of agency 

statistics covers 

50-70% of UIS 

indicators. 

Scope of agency 

statistics covers 

71-90% of UIS 

indicators. 

Scope of agency 

statistics covers 

91-100% of UIS 

indicators. 

100% of OECD indi- 
cators are produced 
by the agency. 


Indicator 3 Classification systems used are broadly consistent with internationally accepted 
standards, guidelines, or good practices. 

Question: Are the classification systems used broadly consistent with international start- 
dards, guidelines, and good practices? 

In general, countries define their own education systems but many of them also generate 
maps that align their own nomenclature with the International Standard Classification 
of Education (ISCED97) (http://www.unesco.org/education/information/nfsunesco/doc/ 
isced_1997.htm). 

• Classification of education is based on UIS ISCED97 and technical guidelines 
and manuals (e.g., level of education, public and private, full-time and part-time, 
trained and untrained). 

• Classification of educational expenditure is based on UIS technical guidelines as 
well as the United Nations System of National Accounts. 

• ISCED and other UIS standards and guidelines are applied consistently to sta- 
tistics on the educational system, students, teachers and educational institutions, 
and educational expenditure. 


Indicator 3 Choose one response in the response matrix and clarify with comments whenever necessary 


Classification sys- 

0 

.25 

.50 

.75 

1 

terns are consistent 

with internation- 
al standards, 
guidelines, or good 
practices. 

ISCED standard is 

not applied. 

Classification is in 

process. 

Classification 
systems are broadly 

consistent with 

international stan- 
dards, guidelines, 
or good practices 
except for students, 
teachers, and 
expenditures. 

Classification 
systems are broadly 

consistent with 

internationally ac- 
cepted standards, 
guidelines, or good 
practices except for 
expenditures. 

Classification sys- 
tems are completely 

consistent with 

internationally ac- 
cepted standards, 
guidelines, or good 
practices. 
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Cover letter and consent to participate 


In advance of the focus group session, participants were provided the following letter as an 
introduction to the proposed session and as a consent form for their signatures. 


HmEL 


August 12, 2014 


Focus Group Informed Consent Form 

In its role as a U.S. funded educational laboratory in support of educational agencies of the 
Pacific Region, McREL International, a nonprofit research organization based in Denver, 
Colorado, is conducting a study of the education management information system (EMIS) 
in the FSM state of Chuuk. You have been asked to take part in this study, which seeks 
to assess and benchmark the quality of this information system in Chuuk. The study will 
be informed by collecting data on a number of indicators that will serve to describe the 
current state of this system. 

Based on your knowledge of the data system in Chuuk, you have been asked to participate 
in a focus group. You have also been asked to provide data codebooks, data collection 
protocols, statistical reports, and policy statements relevant to the indicators that serve to 
benchmark a management information system. Data from your focus group participation 
and the documents you provide will be used to inform the design or redesign of the system. 

Participation in this focus group will last approximately 2 days, about 6 hours per day. The 
conversation will include questions about your judgment regarding aspects of quality in 
the Chuuk EMIS, including statistical policies and practices, data collection procedures, 
utility or serviceability of the statistics, and the presentation and dissemination of findings. 
There are no known risks related to your participation in the focus group. Your participa- 
tion is completely voluntary. You may choose not to answer any particular question. You 
may leave the focus group at any time. 

If you choose to participate in the focus group, notes from the conversation will be cap- 
tured and the conversation may be audio recorded. Although we will not link your name 
with specific comments you make during the focus group, due to the nature of the group 
conversation we are unable to guarantee confidentiality. Elowever, your name will not be 
associated with any statements. Instead, comments will be summarized. We may directly 
quote what is said in a report, but we will not use the name of the person making the 
comment. Any documents you provide will not be attributed to you as the source. Files 
from the focus group will be kept in a safe place during the study and destroyed after the 
end of the audit. Your name will not appear in the report without your consent. 

There are two exceptions to the conditions of privacy discussed in the above paragraph. 
If information is shared about harm or threat of harm to self or others, or child abuse and 
neglect, it is required by law that this be reported to the proper authorities. In addition, 
should any information in this audit be the subject of a court order or lawful subpoena, the 
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research staff may be required to comply with the order or subpoena. However, please be 
assured that the purpose of the focus group is to benchmark current practices in the data 
management system in Chuuk. We do not intend to discuss child abuse, threats, or matters 
of legality. There are no anticipated direct benefits to you for participating in this audit, 
but it should contribute to the improvement of the data management system on Chuuk. 

Should you have any questions about this audit or your rights as a participant, you may call 
Lou Cicchinelli, Senior Fellow, at 303-632-5509 or email at lcicchinelli@mcrel.org. Please 
sign below if you understand and agree to participate. 

I have read (or had someone read) this form and understood the descriptions of the study. 
I have asked for and received a satisfactory explanation of any language that I did not fully 
understand. I agree to participate in this study, and I understand that I may withdraw my 
consent at any time. I have received a copy of this consent form. 


NAME (Please Print) 


Date 
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Appendix D. Participant responses by 
indicator and aspect of system quality 


This appendix includes the average responses of the eight focus group participants for 
each indicator associated with the five aspects of system quality. Some indicators showed 
a range of scores (for example, indicators 1.1 and 1.4 in figure Dl), while others showed 
closer ratings among the data specialists (for example, indicator 1.9 in figure Dl). It is not 
possible to know with any certainty the cause for a score disparity on any one indicator, 
but it is known that the raters came from different perspectives because of their roles and 
responsibilities and, in fact, may have been considering very different aspects of the system 
in their responses. For example, the assessment specialist answers to a different set of data 
requirements and produces different statistical reports than the statistics specialist or data 
clerks. 


D-l 


Figure Dl. Data specialists’ scores for 11 indicators placed prerequisites of quality 
at the established benchmark level 


Score 
1.00 -| 


.80 - 


.60 - 


.40 - 


.20 - 


.00 


1.1 1.2 


<► 


Average for aspect 


♦ Average 
for indicator 
.80-1.00: Mature 
.60-. 79: Established 
.31-. 59: Emerging 
.00-. 30: Latent 


1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 

Indicator 


Note. For descriptions of indicators, see table Dl. 

Source: Authors' calculations based on participant responses to the revised System Assessment and Bench- 
marking for Education Results focus group protocol in 2014. 



Table Dl. Indicators for aspect 1: Prerequisites of quality 

Indicator 

Indicator description 

Legal and institutional environment 

1.1 

The responsibility for collecting, processing, and disseminating statistics is clearly specified. 

1.2 

Data sharing and coordination among data-producing agencies are adequate to facilitate 
data sharing and cooperation between the education statistics agency and other data 
producing agencies. 

1.3 

Respondents’ data are always confidential and used only for statistical purposes. Also, the 
confidentiality of individual respondents’ data is guaranteed and that guarantee is widely 

known. 

1.4 

There is a legal mandate that ensures that individuals give their response to statistical or 
survey questions. 

Human and material resources are adequate for the task 

1.5 

Staff, financial, and computing resources are commensurate with statistical programs of 
the education data agency. 

1.6 

Processes and procedures are in place to ensure that resources are used efficiently. 

1.7 

Education statistics meet user needs and those needs are monitored continuously. 

1.8 

Processes are in place to focus on data quality, on monitoring the quality of the collection, 
processing and dissemination of education statistics, and on the inclusion of data quality 
in statistical planning. 

1.9 

Processes are in place to monitor the quality of the collection, processing, and 

dissemination of statistics. 

1.10 

Processes are in place to deal with quality considerations in planning the statistical program. 

1.11 

Mechanisms exist for addressing new and emerging data requirements. 

Source: World Bank, 2011. 
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Figure D2. Data specialists’ scores for nine indicators placed integrity of education 
statistics at the emerging benchmark level 
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1.00 -| 


.80 - 


.60 - 
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.31-. 59: Emerging 
.00-. 30: Latent 
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Indicator 


2.7 


< ► 


2.8 


<► 


2.9 


Note: Indicators are described in table D2. 

Source: Authors' calculations based on participant responses to the revised System Assessment and Bench- 
marking for Education Results focus group protocol in 2014. 


Table D2. Indicators for aspect 2: Integrity of education statistics 

1 Indicator 

Indicator description p 

Statistical policies and practices are guided by professional principles 

2.1 

Statistics are impartial. Impartiality is assured because the terms and conditions under which 
educational statistics are produced guarantee the professional independence of the agency. 

2.2 

Professionalism is actively promoted and supported within the data-producing agency. 

2.3 

Choices of sources and statistical techniques as well as decisions about disseminations 
are informed solely by statistical considerations. 

2.4 

The appropriate statistical entity is entitled to comment on erroneous interpretation and 
misuse of statistics. 

Statistical policies and practices are transparent 

2.5 

The terms and conditions under which statistics are collected, processed, and 
disseminated are available to the public. 

2.6 

Internal governmental access to statistics prior to their release is publicly identified. 

2.7 

Products of statistical agencies/units are clearly identified as such. 

2.8 

Advance notice is given of major changes in methodology, source data, and statistical 
techniques. 

Policies and practices in education statistics are guided by ethical standards 

2.9 

Guidelines for staff behavior are in place and are well known to the staff. 

Source: World Bank, 2011. 


D-3 






Figure D3. Data specialists’ scores for 10 indicators placed accuracy and reliability 
of education statistics at the established benchmark level 
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3.7 
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Note: Indicators are described in table D3. 

Source: Authors' calculations based on participant responses to the revised System Assessment and Bench- 
marking for Education Results focus group protocol in 2014. 


Table D3. Indicators for aspect 3: Accuracy and reliability of education statistics 

1 Indicator 

Indicator description 9 

3.1 

Source data are obtained from comprehensive data collection programs that take into 
account country-specific conditions. 

3.2 

Source data reasonably approximate the definitions, scope, classifications, valuation, and 
time of recording required. 

3.3 

Source data are timely. 

3.4 

Source data — including censuses, sample surveys, and administrative records — are 
routinely assessed for coverage, sample error, response error, and non-sampling error; the 
results of the assessments are monitored; and corrections to education statistics methods 
are made and published. 

3.5 

Data compilation employs sound statistical techniques to deal with data sources. 

3.6 

Other statistical procedures employ sound statistical techniques. 

3.7 

Assessment and validation of intermediate data and statistical outputs: Intermediate 
results are validated against other information where applicable. 

3.8 

Statistical discrepancies in intermediate data are assessed and investigated. 

3.9 

Statistical discrepancies and other potential indicators of problems in statistical outputs 
are investigated. 

3.10 

Studies and analyses of revisions are carried out routinely and used to inform statistical 

processes. 

Source: World Bank, 2011. 
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Figure D4. Data specialists’ scores for seven indicators placed serviceability at the 
emerging benchmark level 
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Note: Indicators are described in table D4. 

Source: Authors' calculations based on participant responses to the revised System Assessment and Bench- 
marking for Education Results focus group protocol in 2014. 



Table D4. 

Indicators for aspect 4: Serviceability 

1 Indicator 

Indicator description i 

4.1 

Periodicity and timeliness: Periodicity follows dissemination standards. 

4.2 

The timeliness of statistics follows internationally accepted good practices. 

4.3 

Consistency: Statistics are consistent within the dataset. 

4.4 

Statistics are consistent or reconcilable over a reasonable period of time. 

4.5 

Statistics are consistent or reconcilable with those obtained through other data sources 
and/or statistical frameworks. 

4.6 

Revision policy and practice: Revisions follow a regular, well established, and transparent 

schedule. 

4.7 

Preliminary data and/or revised data are clearly identified and revisions are made public. 

Source: World Bank, 2011. 
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Figure D5. Data specialists’ scores for nine indicators placed data accessibility at 
the emerging benchmark level 
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Note: Indicators are described in table D5. 

Source: Authors' calculations based on participant responses to the revised System Assessment and Bench- 
marking for Education Results focus group protocol in 2014. 



Table D5. 

Indicators for aspect 5: Accessibility 

1 Indicator 

Indicator description g 

5.1 

Statistics are presented in a way that facilitates proper interpretation and meaningful 
comparisons (layout and clarity of text, tables, and charts). 

5.2 

Dissemination media and formats are adequate. 

5.3 

Statistics are released on a pre-announced schedule. 

5.4 

Statistics are made available to all users at the same time. 

5.5 

Statistics not routinely disseminated are made available upon request. 

5.6 

Metadata accessibility: All metadata documentation is available, and differences from 

international standards are annotated. 

5.7 

Levels of detail are adapted to the needs of the intended audience. 

5.8 

Assistance to users: Contact points for each subject field are publicized. 

5.9 

Catalogues of publications, documents, and other services, including information on any 
charges, are widely available. 

Source: World Bank, 2011. 
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Note 


1. This report documents a study that was replicated in the four states of the Federated 
States of Micronesia: Chuuk, Kosrae, Pohnpei, and Yap. Thus, although the data and 
findings naturally differ from one report to the other, the explanation of why the study 
was done, the review of relevant literature, and the description of methods are virtual- 
ly the same in each report. 


Note-1 
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